diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 5d7985c752b..0d8439c78f3 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -81,6 +81,7 @@ /packages/azure/data_stream/activitylogs @elastic/obs-infraobs-integrations /packages/azure/data_stream/auditlogs @elastic/obs-infraobs-integrations /packages/azure/data_stream/eventhub @elastic/obs-ds-hosted-services +/packages/azure/data_stream/events @elastic/obs-ds-hosted-services /packages/azure/data_stream/identity_protection @elastic/obs-infraobs-integrations /packages/azure/data_stream/platformlogs @elastic/obs-infraobs-integrations /packages/azure/data_stream/provisioning @elastic/obs-infraobs-integrations diff --git a/packages/azure/_dev/build/docs/events.md b/packages/azure/_dev/build/docs/events.md new file mode 100644 index 00000000000..f3ebfd70006 --- /dev/null +++ b/packages/azure/_dev/build/docs/events.md @@ -0,0 +1,504 @@ +# Azure Logs Integration (v2 preview) + +The Azure Logs integration (v2 preview) collects logs from selected Azure services, such as Microsoft Entra ID (Sign-in, Audit, Identity Protection, and Provisioning logs), Azure Spring Apps, Azure Firewall, Microsoft Graph Activity, and several others. + +You can then visualize that data in Kibana, create alerts if something goes wrong, and reference data when troubleshooting an issue. + +For example, to detect possible brute force sign-in attacks, you can install the Azure Logs integration to send Azure sign-in logs to Elastic. Then, by setting up a new rule in the Elastic Observability Logs app, you can be alerted when the number of failed sign-in attempts exceeds a certain threshold. + +You may also want to plan your Azure capacity better. Send Azure Activity logs to Elastic to track and visualize when your virtual machines fail to start because they exceed the quota limit. + +## What's new in v2 preview? + +The Azure Logs integration (v2 preview) introduces a new architecture that allows you to forward logs from multiple Azure services to the same event hub. + +```text + ┌─────────────────┐ + │ activity logs │ + ┌─▶│ <> │ + │ └─────────────────┘ + │ +┌───────────────┐ ┌─────────────┐ ┌─────────────────┐ │ ┌─────────────────┐ +│ logs │ │ Elastic │ │ events (router) │ │ │ firewall logs │ +│ <> │──▶│ Agent │─▶│ <> │──┼─▶│ <> │ +└───────────────┘ └─────────────┘ └─────────────────┘ │ └─────────────────┘ + │ + │ ┌─────────────────┐ + │ │ signin logs │ + └─▶│ <> │ + └─────────────────┘ +``` + +The integration will automatically detect the log category and forward the logs to the appropriate data stream. When the integration v2 preview cannot find a matching data stream for a log category, it forwards the logs to the platform logs data stream. + +IMPORTANT: To use the v2 preview, you must turn off all the existing v1 integrations and turn on only the v2 preview integration. + +Under the hood, the v2 preview uses only one `azure-eventhub` input per event hub. The v2 preview avoids contention and inefficiencies from using multiple inputs with the same event hub, which is typical of the v1 architecture. With the v2 preview, you can still assign the agent policy to multiple Elastic Agents to scale out the logs processing. + +## Data streams + +The Azure Logs integration (v2 preview) collects logs. + +**Logs** help you keep a record of events that happen on your Azure account. +Log data streams collected by the Azure Logs integration include Activity, Platform, Microsoft Entra ID (Sign-in, Audit, Identity Protection, Provisioning), Microsoft Graph Activity, and Spring Apps logs. + +## Requirements + +You need Elasticsearch to store and search for your data and Kibana to visualize and manage it. +You can use our recommended hosted Elasticsearch Service on Elastic Cloud or self-manage the Elastic Stack on your hardware. + +Before using the Azure integration, you will need: + +* One or more **diagnostic settings** to export logs from Azure services to Event Hubs. +* One **event hub** to store in-flight logs exported by Azure services and make them available to Elastic Agent. +* One **Storage Account container** to store the event hub checkpointing information for each partition. + +### Diagnostic settings + +Azure diagnostic settings allow you to export metrics and logs from a **source** service (or resource) to one **destination** for analysis and long-term storage. + +```text + ┌────────────────────┐ ┌──────────────┐ ┌─────────────────┐ + │ Microsoft Entra ID │ │ Diagnostic │ │ Event Hub │ + │ <> │─────▶│ settings │────▶│ <> │ + └────────────────────┘ └──────────────┘ └─────────────────┘ +``` + +Examples of source services: + +* Azure Monitor +* Microsoft Entra ID +* Azure Firewall + +The diagnostic settings support several destination types. The Elastic Agent requires diagnostic settings configured with an event hub as the destination. + +### Event Hub + +[Azure Event Hubs](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-about) is a data streaming platform and event ingestion service that can receive and temporarily store millions of events. + +Elastic Agent with the Azure Logs integration will consume logs published in the Event Hubs service. + +```text + ┌────────────────┐ ┌────────────┐ + │ adlogs │ │ Elastic │ + │ <> │─────▶│ Agent │ + └────────────────┘ └────────────┘ +``` + +To learn more about Event Hubs, refer to [Features and terminology in Azure Event Hubs](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-features). + +### Storage account container + +The [Storage account](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview) is a versatile Azure service that allows you to store data in various storage types, including blobs, file shares, queues, tables, and disks. + +The Azure Logs integration requires a Storage Account container to work. The integration uses the Storage Account container for checkpointing; it stores data about the Consumer Group (state, position, or offset) and shares it among the Elastic Agents. Sharing such information allows multiple Elastic Agents assigned to the same agent policy to work together, enabling horizontal scaling of the logs processing when required. + +```text + ┌────────────────┐ ┌────────────┐ + │ adlogs │ logs │ Elastic │ + │ <> │────────────────────▶│ Agent │ + └────────────────┘ └────────────┘ + │ + consumer group info │ + ┌────────────────┐ (state, position, or │ + │ azurelogs │ offset) │ + │ <> │◀───────────────────────────┘ + └────────────────┘ +``` + +The Elastic Agent automatically creates one container for each enabled integration. In the container, the Agent will create one blob for each existing partition on the event hub. + +For example, if you enable one integration to fetch data from an event hub with four partitions, the Agent will create the following: + +* One Storage Account container. +* Four blobs in that container. + +The information stored in the blobs is small (usually < 300 bytes per blob) and accessed relatively frequently. Elastic recommends using the Hot storage tier. + +You need to keep the Storage Account container as long as you need to run the integration with the Elastic Agent. If you delete a storage account container, the Elastic Agent will stop working and create a new one the next time it starts. By deleting a storage account container, the Elastic Agent will lose track of the last message processed and start processing messages from the beginning of the event hub retention period. + +## Setup + +With the Azure Logs integration (v2 preview), you can forward logs from multiple Azure services to the same event hub. The integration will automatically detect the log category and forward the logs to the appropriate data stream. + +```text + ┌─────────────────┐ + │ activity logs │ + ┌─▶│ <> │ + │ └─────────────────┘ + │ +┌───────────────┐ ┌─────────────┐ ┌─────────────────┐ │ ┌─────────────────┐ +│ logs │ │ Elastic │ │ events (router) │ │ │ firewall logs │ +│ <> │──▶│ Agent │─▶│ <> │──┼─▶│ <> │ +└───────────────┘ └─────────────┘ └─────────────────┘ │ └─────────────────┘ + │ + │ ┌─────────────────┐ + │ │ signin logs │ + └─▶│ <> │ + └─────────────────┘ +``` + +Before adding the integration, you must complete the following tasks. + +### Create an Event Hub + +The event hub receives the logs exported from the Azure service and makes them available for the Elastic Agent to read. + +Here's a high-level overview of the required steps: + +* Create a resource group, or select an existing one. +* Create an Event Hubs namespace. +* Create an event hub. + +For a detailed step-by-step guide, check the quickstart [Create an event hub using Azure portal](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-create). + +Take note of the event hub **Name**, which you will use later when specifying an **eventhub** in the integration settings. + +#### Event Hubs Namespace vs Event Hub + +In the integration settings, you should use the event hub name (not the Event Hubs namespace name) as the value for the **event hub ** option. + +If you are new to Event Hubs, think of the Event Hubs namespace as the cluster and the event hub as the topic. You will typically have one cluster and multiple topics. + +If you are familiar with Kafka, here's a conceptual mapping between the two: + +| Kafka Concept | Event Hub Concept | +|----------------|-------------------| +| Cluster | Namespace | +| Topic | An event hub | +| Partition | Partition | +| Consumer Group | Consumer Group | +| Offset | Offset | + +#### How many partitions? + +The number of partitions is essential to balance the event hub cost and performance. + +Here are a few examples with one or multiple agents, with recommendations on picking the correct number of partitions for your use case. + +##### Single Agent + +With a single Agent deployment, increasing the number of partitions on the event hub is the primary driver in scale-up performances. The Agent creates one worker for each partition. + +```text +┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ + +│ │ │ │ + +│ ┌─────────────────┐ │ │ ┌─────────────────┐ │ + │ partition 0 │◀───────────│ consumer │ +│ └─────────────────┘ │ │ └─────────────────┘ │ + ┌─────────────────┐ ┌─────────────────┐ +│ │ partition 1 │◀──┼────┼───│ consumer │ │ + └─────────────────┘ └─────────────────┘ +│ ┌─────────────────┐ │ │ ┌─────────────────┐ │ + │ partition 2 │◀────────── │ consumer │ +│ └─────────────────┘ │ │ └─────────────────┘ │ + ┌─────────────────┐ ┌─────────────────┐ +│ │ partition 3 │◀──┼────┼───│ consumer │ │ + └─────────────────┘ └─────────────────┘ +│ │ │ │ + +│ │ │ │ + +└ Event Hub ─ ─ ─ ─ ─ ─ ─ ┘ └ Agent ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ +``` + +##### Two or more Agents + +With more than one Agent, setting the number of partitions is crucial. The agents share the existing partitions to scale out performance and improve availability. + +The number of partitions must be at least the number of agents. + +```text +┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ + +│ │ │ ┌─────────────────┐ │ + ┌──────│ consumer │ +│ ┌─────────────────┐ │ │ │ └─────────────────┘ │ + │ partition 0 │◀────┘ ┌─────────────────┐ +│ └─────────────────┘ │ ┌──┼───│ consumer │ │ + ┌─────────────────┐ │ └─────────────────┘ +│ │ partition 1 │◀──┼─┘ │ │ + └─────────────────┘ ─Agent─ ─ ─ ─ ─ ─ ─ ─ ─ ─ +│ ┌─────────────────┐ │ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ + │ partition 2 │◀────┐ +│ └─────────────────┘ │ │ │ ┌─────────────────┐ │ + ┌─────────────────┐ └─────│ consumer │ +│ │ partition 3 │◀──┼─┐ │ └─────────────────┘ │ + └─────────────────┘ │ ┌─────────────────┐ +│ │ └──┼──│ consumer │ │ + └─────────────────┘ +│ │ │ │ + +└ Event Hub ─ ─ ─ ─ ─ ─ ─ ┘ └ Agent ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ +``` + +##### Recommendations + +Create an event hub with at least two partitions. Two partitions allow low-volume deployment to support high availability with two agents. Consider creating four partitions or more to handle medium-volume deployments with availability. + +To learn more about event hub partitions, read an in-depth guide from Microsoft at [Quickstart: Create an event hub using Azure portal](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-create). + +To learn more about event hub partition from the performance perspective, check the scalability-focused document at [Event Hubs scalability](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-scalability#partitions). + +#### How many Event Hubs? + +With the Azure Logs integration (v2 preview), Elastic strongly recommends creating one event hub and using it for all Azure services. + +For example, if you plan to collect Microsoft Entra ID and Azure Firewall logs, create one event hub and use it for both services. + +Here's a high-level diagram of the solution: + +```text +┌────────────────┐ ┌───────────────┐ +│ MS Entra ID │ │ Diagnostic │ +│ <> │──▶│ Settings │─┐ +└────────────────┘ └───────────────┘ │ + │ ┌───────────────┐ ┌─────────────┐ + │ │ logs │ │ Elastic │ + ├─▶│ <> │──▶│ Agent │ + │ └───────────────┘ └─────────────┘ +┌────────────────┐ ┌───────────────┐ │ +│ Azure Firewall │ │ Diagnostic │ │ +│ <> │──▶│ Settings │─┘ +└────────────────┘ └───────────────┘ +``` + +The Azure Logs integration (v2 preview) will automatically detect the log category and forward the logs to the appropriate data stream. + +#### Consumer Group + +Like all other event hub clients, Elastic Agent needs a consumer group name to access the event hub. + +A Consumer Group is an entire event hub's view (state, position, or offset). Consumer groups enable multiple agents to have a separate view of the event stream and to read the logs independently at their own pace and with their offsets. + +Consumer groups allow multiple Elastic Agents assigned to the same agent policy to work together; this enables horizontal scaling of the logs processing when required. + +In most cases, you can use the default consumer group named `$Default`. If `$Default` is already used by other applications, you can create a consumer group dedicated to the Azure Logs integration. + +#### Connection string + +The Elastic Agent requires a connection string to access the event hub and fetch the exported logs. The connection string contains details about the event hub used and the credentials required to access it. + +To get the connection string for your Event Hubs namespace: + +1. Visit the **Event Hubs namespace** you created in a previous step. +1. Select **Settings** > **Shared access policies**. + +Create a new Shared Access Policy (SAS): + +1. Select **Add** to open the creation panel. +1. Add a **Policy name** (for example, "ElasticAgent"). +1. Select the **Listen** claim. +1. Select **Create**. + +When the SAS Policy is ready, select it to display the information panel. + +Take note of the **Connection string–primary key**, which you will use later when specifying a **connection_string** in the integration settings. + +### Create a diagnostic settings + +The diagnostic settings export the logs from Azure services to a destination and in order to use Azure Logs integration, it must be an event hub. + +To create a diagnostic settings to export logs: + +1. Locate the diagnostic settings for the service (for example, Microsoft Entra ID). +2. Select diagnostic settings in the **Monitoring** section of the service. Note that different services may place the diagnostic settings in various positions. +3. Select **Add diagnostic settings**. + +In the diagnostic settings page you must select the source **log categories** you want to export and then select their **destination**. + +#### Select log categories + +Each Azure service exports a well-defined list of log categories. Check the individual integration doc to learn which log categories the integration supports. + +#### Select the destination + +Select the **subscription** and the **Event Hubs namespace** you previously created. Select the event hub dedicated to this integration. + +```text + ┌───────────────┐ ┌──────────────┐ ┌───────────────┐ ┌───────────┐ + │ MS Entra ID │ │ Diagnostic │ │ adlogs │ │ Elastic │ + │ <> ├──▶│ Settings │──▶│ <> │─────▶│ Agent │ + └───────────────┘ └──────────────┘ └───────────────┘ └───────────┘ +``` + +### Create a Storage Account container + +The Elastic Agent stores the event hub checkpoint information in a storage account container. Storing checkpoint information in a container allows agents to share message processing and resume from the last processed message after a restart. + +**Note**: Use the Storage Account as a checkpoint store only. + +To create the storage account: + +1. Sign in to the [Azure Portal](https://portal.azure.com/) and create your storage account. +2. While configuring your project details, make sure you select the following recommended default settings: + * Hierarchical namespace: disabled + * Minimum TLS version: Version 1.2 + * Access tier: Hot + * Enable soft delete for blobs: disabled + * Enable soft delete for containers: disabled + +3. When the new storage account is ready, you need to take note of the storage account name and the Storage Account access keys, as you will use them later to authenticate your Elastic application’s requests to this storage account. + +This is the final diagram of the setup for collecting Activity logs from the Azure Monitor service. + +```text + ┌───────────────┐ ┌──────────────┐ ┌────────────────┐ ┌───────────┐ + │ MS Entra ID │ │ Diagnostic │ │ adlogs │ logs │ Elastic │ + │ <> ├──▶│ Settings │──▶│ <> │────────▶│ Agent │ + └───────────────┘ └──────────────┘ └────────────────┘ └───────────┘ + │ + ┌──────────────┐ consumer group info │ + │ azurelogs │ (state, position, or │ + │<> │◀───────────────offset)──────────────┘ + └──────────────┘ +``` + +#### How many Storage Accounts? + +The Elastic Agent can use a single Storage Account to store the checkpoint information for multiple integrations. + +**CRITICAL**: make sure to use a different **storage_account_container** for each integration. The Elastic Agent uses the **integration name** and the **event hub name** to uniquely identify the container that holds the blobs with the checkpoint information. + +```text +┌─────────────────────────────────┐ ┌──────────────────────────────────────────┐ +│ │ │ │ +│ ┌─────────────────────┐ │ │ ┌────────────────────────────────────┐ │ +│ │ azure-eventhub │ │ │ │ filebeat-activitylogs-eventhub-1 │ │ +│ │ <> │──────┼──────┼─▶│ <> │ │ +│ └─────────────────────┘ │ │ └────────────────────────────────────┘ │ +│ ┌─────────────────────┐ │ │ ┌────────────────────────────────────┐ │ +│ │ azure-eventhub │ │ │ │ filebeat-signinlogs-eventhub-2 │ │ +│ │ <> │──────┼──────┼─▶│ <> │ │ +│ └─────────────────────┘ │ │ └────────────────────────────────────┘ │ +│ ┌─────────────────────┐ │ │ ┌────────────────────────────────────┐ │ +│ │ azure-eventhub │ │ │ │ filebeat-auditlogs-eventhub-3 │ │ +│ │ <> │──────┼──────┼─▶│ <> │ │ +│ └─────────────────────┘ │ │ └────────────────────────────────────┘ │ +│ │ │ │ +└─Elastic Agent───────────────────┘ └─Storage Account──────────────────────────┘ +``` + +### Running the integration behind a firewall + +When you run the Elastic Agent behind a firewall, you must allow traffic on ports `5671` and `5672` for the event hub and port `443` for the Storage Account container to ensure proper communication with the necessary components. + +```text +┌────────────────────────────────┐ ┌───────────────────┐ ┌───────────────────┐ +│ │ │ │ │ │ +│ ┌────────────┐ ┌───────────┐ │ │ ┌──────────────┐ │ │ ┌───────────────┐ │ +│ │ diagnostic │ │ event hub │ │ │ │azure-eventhub│ │ │ │ activity logs │ │ +│ │ setting │──▶│ │◀┼AMQP─│ <> │─┼──┼▶│<>│ │ +│ └────────────┘ └───────────┘ │ │ └──────────────┘ │ │ └───────────────┘ │ +│ │ │ │ │ │ │ +│ │ │ │ │ │ │ +│ │ │ │ │ │ │ +│ ┌─────────────┬─────HTTPS─┼──────────┘ │ │ │ +│ ┌───────┼─────────────┼──────┐ │ │ │ │ │ +│ │ │ │ │ │ │ │ │ │ +│ │ ▼ ▼ │ │ └─Agent─────────────┘ └─Elastic Cloud─────┘ +│ │ ┌──────────┐ ┌──────────┐ │ │ +│ │ │ 0 │ │ 1 │ │ │ +│ │ │ <> │ │ <> │ │ │ +│ │ └──────────┘ └──────────┘ │ │ +│ │ │ │ +│ │ │ │ +│ └─Storage Account Container──┘ │ +│ │ +│ │ +└─Azure──────────────────────────┘ +``` + +#### Event Hub + +Port `5671` and `5672` are commonly used for secure communication with the event hub. These ports are used to receive events. The Elastic Agent can establish a secure connection with the event hub by allowing traffic on these ports. + +For more information, check the following documents: + +* [What ports do I need to open on the firewall?](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-faq#what-ports-do-i-need-to-open-on-the-firewall) from the [Event Hubs frequently asked questions](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-faq#what-ports-do-i-need-to-open-on-the-firewall). +* [AMQP outbound port requirements](https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-protocol-guide#amqp-outbound-port-requirements) + +#### Storage Account container + +The Elastic Agent uses port `443` for secure communication with the Storage Account container. By allowing traffic on port 443, the Elastic Agent can securely access and interact with the Storage Account container, essential for storing and retrieving checkpoint data for each event hub partition. + +#### DNS + +Optionally, you can restrict the traffic to the following domain names: + +```text +*.servicebus.windows.net +*.blob.core.windows.net +*.cloudapp.net +``` + +## Settings + +Use the following settings to configure the Azure Logs integration when you add it to Fleet. + +`eventhub` : +_string_ +A fully managed, real-time data ingestion service. Elastic recommends using only letters, numbers, and the hyphen (-) character for event hub names to maximize compatibility. You can use existing event hubs having underscores (_) in the event hub name; in this case, the integration will replace underscores with hyphens (-) when it uses the event hub name to create dependent Azure resources behind the scenes (e.g., the storage account container to store event hub consumer offsets). Elastic also recommends using a separate event hub for each log type as the field mappings of each log type differ. +Default value `insights-operational-logs`. + +`consumer_group` : +_string_ +Enable the publish/subscribe mechanism of Event Hubs with consumer groups. A consumer group is a view (state, position, or offset) of an entire event hub. Consumer groups enable multiple consuming applications to each have a separate view of the event stream, and to read the stream independently at their own pace and with their own offsets. +Default value: `$Default` + +`connection_string` : +_string_ + +The connection string is required to communicate with Event Hubs. See [Get an Event Hubs connection string](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string) for more information. + +A Blob Storage account is required to store/retrieve/update the offset or state of the event hub messages. This allows the integration to start back up when it stopped processing messages. + +`storage_account` : +_string_ +The name of the storage account that the state/offsets will be stored and updated. + +`storage_account_key` : +_string_ +The storage account key. Used to authorize access to data in your storage account. + +`storage_account_container` : +_string_ +The storage account container where the integration stores the checkpoint data for the consumer group. It is an advanced option to use with extreme care. You MUST use a dedicated storage account container for each Azure log type (activity, sign-in, audit logs, and others). DO NOT REUSE the same container name for more than one Azure log type. See [Container Names](https://docs.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata#container-names) for details on naming rules from Microsoft. The integration generates a default container name if not specified. + +`resource_manager_endpoint` : +_string_ +Optional. By default, the integration uses the Azure public environment. To override this and use a different Azure environment, users can provide a specific resource manager endpoint + +Examples: + +* Azure ChinaCloud: `https://management.chinacloudapi.cn/` +* Azure GermanCloud: `https://management.microsoftazure.de/` +* Azure PublicCloud: `https://management.azure.com/` +* Azure USGovernmentCloud: `https://management.usgovcloudapi.net/` + +This setting can also be used to define your own endpoints, like for hybrid cloud models. + +## Handling Malformed JSON in Azure Logs + +Azure services have been observed occasionally sending [malformed JSON](https://learn.microsoft.com/en-us/answers/questions/1001797/invalid-json-logs-produced-for-function-apps) documents. These logs can disrupt the expected JSON formatting and lead to parsing issues during processing. + +To address this issue, the advanced settings section of each data stream offers two sanitization options: + +* Sanitizes New Lines: removes new lines in logs. +* Sanitizes Single Quotes: replace single quotes with double quotes in logs, excluding single quotes occurring within double quotes. + +Malformed logs can be identified by: + +* The presence of a records array in the message field indicates a failure to unmarshal the byte slice. +* An `error.message` field contains the "Received invalid JSON from the Azure Cloud platform. Unable to parse the source log message" text. + +Known data streams that might produce malformed logs: + +* Platform Logs +* Spring Apps Logs + +## Reference + +Visit the page for each individual Azure Logs integration to see details about exported fields and sample events. diff --git a/packages/azure/changelog.yml b/packages/azure/changelog.yml index bd20d10043f..ce1de92d67f 100644 --- a/packages/azure/changelog.yml +++ b/packages/azure/changelog.yml @@ -1,3 +1,8 @@ +- version: "1.20.0" + changes: + - description: Add Azure Logs integration v2 (preview) + type: enhancement + link: https://github.com/elastic/integrations/pull/11984 - version: "1.19.4" changes: - description: Fix destination.geo.region_name mapping. diff --git a/packages/azure/data_stream/events/agent/stream/stream.yml.hbs b/packages/azure/data_stream/events/agent/stream/stream.yml.hbs new file mode 100644 index 00000000000..f9a2b33ba72 --- /dev/null +++ b/packages/azure/data_stream/events/agent/stream/stream.yml.hbs @@ -0,0 +1,46 @@ +{{#if connection_string}} +connection_string: {{connection_string}} +{{/if}} +{{#if storage_account_container }} +storage_account_container: {{storage_account_container}} +{{else}} +{{#if eventhub}} +storage_account_container: filebeat-events-{{eventhub}} +{{/if}} +{{/if}} +{{#if eventhub}} +eventhub: {{eventhub}} +{{/if}} +{{#if consumer_group}} +consumer_group: {{consumer_group}} +{{/if}} +{{#if storage_account}} +storage_account: {{storage_account}} +{{/if}} +{{#if storage_account_key}} +storage_account_key: {{storage_account_key}} +{{/if}} +{{#if resource_manager_endpoint}} +resource_manager_endpoint: {{resource_manager_endpoint}} +{{/if}} +tags: +{{#if preserve_original_event}} + - preserve_original_event +{{/if}} +{{#each tags as |tag i|}} + - {{tag}} +{{/each}} +{{#contains "forwarded" tags}} +publisher_pipeline.disable_host: true +{{/contains}} +{{#if processors}} +processors: +{{processors}} +{{/if}} +sanitize_options: +{{#if sanitize_newlines}} + - NEW_LINES +{{/if}} +{{#if sanitize_singlequotes}} + - SINGLE_QUOTES +{{/if}} diff --git a/packages/azure/data_stream/events/elasticsearch/ingest_pipeline/default.yml b/packages/azure/data_stream/events/elasticsearch/ingest_pipeline/default.yml new file mode 100644 index 00000000000..98a83b216b2 --- /dev/null +++ b/packages/azure/data_stream/events/elasticsearch/ingest_pipeline/default.yml @@ -0,0 +1,96 @@ +--- +description: Pipeline for parsing Azure logs. +processors: + - set: + field: ecs.version + value: '8.11.0' + # TODO: we can remove this processor when https://github.com/elastic/beats/issues/40561 + # is fixed and released. + - rename: + field: azure + target_field: azure-eventhub + if: 'ctx.azure?.eventhub != null' + ignore_missing: true + - set: + field: event.kind + value: event + + # + # Set `event.dataset` value based on the event category. + # ------------------------------------------------------ + # + # In the `routing_rules.yml` file, we use the + # `event.dataset` value to route the event to + # the appropriate data stream. + # + + - json: + field: message + target_field: tmp_json + description: 'Parses the message field as JSON and stores it in a temporary field to identify the event dataset.' + + # Defaults to azure.events if the `category` field is not present. + - set: + field: event.dataset + value: azure.events + description: 'Sets the default event dataset.' + + # Sets the event dataset based on the `category` field. + - set: + field: event.dataset + value: azure.platformlogs + if: 'ctx.tmp_json?.category != null' + description: 'If the event has a category field, we consider it a platform log.' + - set: + field: event.dataset + value: azure.activitylogs + if: 'ctx.tmp_json?.category == "Administrative" || ctx.tmp_json?.category == "Security" || ctx.tmp_json?.category == "ServiceHealth" || ctx.tmp_json?.category == "Alert" || ctx.tmp_json?.category == "Recommendation" || ctx.tmp_json?.category == "Policy" || ctx.tmp_json?.category == "Autoscale" || ctx.tmp_json?.category == "ResourceHealth"' + - set: + field: event.dataset + value: azure.application_gateway + if: 'ctx.tmp_json?.category == "ApplicationGatewayFirewallLog" || ctx.tmp_json?.category == "ApplicationGatewayAccessLog"' + - set: + field: event.dataset + value: azure.auditlogs + if: 'ctx.tmp_json?.category == "AuditLogs"' + - set: + field: event.dataset + value: azure.firewall_logs + if: 'ctx.tmp_json?.category == "AzureFirewallApplicationRule" || ctx.tmp_json?.category == "AzureFirewallNetworkRule" || ctx.tmp_json?.category == "AzureFirewallDnsProxy" || ctx.tmp_json?.category == "AZFWApplicationRule" || ctx.tmp_json?.category == "AZFWNetworkRule" || ctx.tmp_json?.category == "AZFWNatRule" || ctx.tmp_json?.category == "AZFWDnsQuery"' + - set: + field: event.dataset + value: azure.graphactivitylogs + if: 'ctx.tmp_json?.category == "MicrosoftGraphActivityLogs"' + - set: + field: event.dataset + value: azure.identity_protection + if: 'ctx.tmp_json?.category == "RiskyUsers" || ctx.tmp_json?.category == "UserRiskEvents"' + - set: + field: event.dataset + value: azure.provisioning + if: 'ctx.tmp_json?.category == "ProvisioningLogs"' + - set: + field: event.dataset + value: azure.signinlogs + if: 'ctx.tmp_json?.category == "SignInLogs" || ctx.tmp_json?.category == "NonInteractiveUserSignInLogs" || ctx.tmp_json?.category == "ServicePrincipalSignInLogs" || ctx.tmp_json?.category == "ManagedIdentitySignInLogs"' + - set: + field: event.dataset + value: azure.springcloudlogs + if: 'ctx.tmp_json?.category == "ApplicationConsole" || ctx.tmp_json?.category == "SystemLogs" || ctx.tmp_json?.category == "IngressLogs" || ctx.tmp_json?.category == "BuildLogs" || ctx.tmp_json?.category == "ContainerEventLogs"' + description: 'Azure Spring Apps log categories (refs: https://learn.microsoft.com/en-us/azure/azure-monitor/reference/supported-logs/microsoft-appplatform-spring-logs)' + + # Remove the temporary field used to identify the event dataset. + - remove: + field: tmp_json + ignore_missing: true + description: 'Removes the temporary field used to identify the event dataset.' + +# +# +# Error handling +# + +on_failure: + - set: + field: error.message + value: '{{ _ingest.on_failure_message }}' diff --git a/packages/azure/data_stream/events/fields/base-fields.yml b/packages/azure/data_stream/events/fields/base-fields.yml new file mode 100644 index 00000000000..5ec2a7e7a1c --- /dev/null +++ b/packages/azure/data_stream/events/fields/base-fields.yml @@ -0,0 +1,16 @@ +- name: '@timestamp' + type: date + description: Event timestamp. +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset name. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: event.module + type: constant_keyword + description: Event module + value: azure diff --git a/packages/azure/data_stream/events/fields/fields.yml b/packages/azure/data_stream/events/fields/fields.yml new file mode 100644 index 00000000000..e8d00807f3d --- /dev/null +++ b/packages/azure/data_stream/events/fields/fields.yml @@ -0,0 +1,27 @@ +- name: azure-eventhub + type: group + fields: + - name: eventhub + type: keyword + description: | + Event hub name + - name: offset + type: long + description: | + Offset + - name: enqueued_time + type: keyword + description: | + The enqueued time + - name: partition_id + type: keyword + description: | + Partition ID + - name: consumer_group + type: keyword + description: | + Consumer group + - name: sequence_number + type: long + description: |- + Sequence number diff --git a/packages/azure/data_stream/events/fields/package-fields.yml b/packages/azure/data_stream/events/fields/package-fields.yml new file mode 100644 index 00000000000..58d74208552 --- /dev/null +++ b/packages/azure/data_stream/events/fields/package-fields.yml @@ -0,0 +1,42 @@ +- name: azure + type: group + fields: + - name: subscription_id + type: keyword + description: | + Azure subscription ID + - name: correlation_id + type: keyword + description: | + Correlation ID + - name: tenant_id + type: keyword + description: | + Tenant ID + - name: resource + type: group + fields: + - name: id + type: keyword + description: | + Resource ID + - name: group + type: keyword + description: | + Resource group + - name: provider + type: keyword + description: | + Resource type/namespace + - name: namespace + type: keyword + description: | + Resource type/namespace + - name: name + type: keyword + description: | + Name + - name: authorization_rule + type: keyword + description: | + Authorization rule diff --git a/packages/azure/data_stream/events/manifest.yml b/packages/azure/data_stream/events/manifest.yml new file mode 100644 index 00000000000..f99aac1c229 --- /dev/null +++ b/packages/azure/data_stream/events/manifest.yml @@ -0,0 +1,86 @@ +type: logs +title: Azure Logs (v2 preview) +dataset: azure.events +streams: + - input: "azure-eventhub" + enabled: false + template_path: "stream.yml.hbs" + title: Collect Azure logs from Event Hub + description: | + + Collect all the supported (see list below) Azure logs from Event Hub to a target data stream. + + ✨ **New in version 1.20.0+:** by enabling this integration, you can collect all the logs from the following Azure services and route them to the appropriate data stream: + + - Microsoft Entra ID logs: + - Audit + - Identity Protection + - Provisioning + - Sign-in + - Platform logs + - Activity logs + - Microsoft Graph Activity Logs + - Spring Apps logs + - Firewall logs + - Application Gateway logs + + **You MUST turn off the v1 integrations** when you enable this v2 integration. If you run both integrations simultaneously, you will see duplicate logs in your data stream. + + If you need to collect raw events from Azure Event Hub, we recommend using the [Custom Azure Logs integration](https://www.elastic.co/docs/current/integrations/azure_logs) which provides more flexibility. + + To learn more about the efficiency and routing enhancements introduced in version 1.20.0, please read the [Azure Logs (v2 preview)](https://www.elastic.co/docs/current/integrations/azure/events) documentation. + + vars: + - name: preserve_original_event + required: true + show_user: true + title: Preserve original event + description: Preserves a raw copy of the original event, added to the field `event.original` + type: bool + multi: false + default: false + - name: storage_account_container + type: text + title: Storage Account Container + multi: false + required: false + show_user: false + description: > + The storage account container where the integration stores the checkpoint data for the consumer group. It is an advanced option to use with extreme care. You MUST use a dedicated storage account container for each Azure log type (activity, sign-in, audit logs, and others). DO NOT REUSE the same container name for more than one Azure log type. See [Container Names](https://docs.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata#container-names) for details on naming rules from Microsoft. The integration generates a default container name if not specified. + - name: tags + type: text + title: Tags + multi: true + required: true + show_user: false + default: + - azure-eventhub + - forwarded + - name: processors + type: yaml + title: Processors + multi: false + required: false + show_user: false + description: > + Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details. + - name: sanitize_newlines + type: bool + title: Sanitizes New Lines + description: Removes new lines in logs to ensure proper formatting of JSON data and avoid parsing issues during processing. + multi: false + required: false + show_user: false + default: false + - name: sanitize_singlequotes + required: true + show_user: false + title: Sanitizes Single Quotes + description: Replaces single quotes with double quotes (single quotes inside double quotes are omitted) in logs to ensure proper formatting of JSON data and avoid parsing issues during processing. + type: bool + multi: false + default: false +# Ensures agents have permissions to write data to `logs-*-*` +elasticsearch: + dynamic_dataset: true + dynamic_namespace: true diff --git a/packages/azure/data_stream/events/routing_rules.yml b/packages/azure/data_stream/events/routing_rules.yml new file mode 100644 index 00000000000..e204a0a5aad --- /dev/null +++ b/packages/azure/data_stream/events/routing_rules.yml @@ -0,0 +1,57 @@ +- source_dataset: azure.events + rules: + - target_dataset: azure.activitylogs + if: ctx.event?.dataset == 'azure.activitylogs' + namespace: + - "{{data_stream.namespace}}" + - default + - target_dataset: azure.application_gateway + if: ctx.event?.dataset == 'azure.application_gateway' + namespace: + - "{{data_stream.namespace}}" + - default + - target_dataset: azure.auditlogs + if: ctx.event?.dataset == 'azure.auditlogs' + namespace: + - "{{data_stream.namespace}}" + - default + - target_dataset: azure.eventhub + if: ctx.event?.dataset == 'azure.eventhub' + namespace: + - "{{data_stream.namespace}}" + - default + - target_dataset: azure.firewall_logs + if: ctx.event?.dataset == 'azure.firewall_logs' + namespace: + - "{{data_stream.namespace}}" + - default + - target_dataset: azure.graphactivitylogs + if: ctx.event?.dataset == 'azure.graphactivitylogs' + namespace: + - "{{data_stream.namespace}}" + - default + - target_dataset: azure.identity_protection + if: ctx.event?.dataset == 'azure.identity_protection' + namespace: + - "{{data_stream.namespace}}" + - default + - target_dataset: azure.platformlogs + if: ctx.event?.dataset == 'azure.platformlogs' + namespace: + - "{{data_stream.namespace}}" + - default + - target_dataset: azure.provisioning + if: ctx.event?.dataset == 'azure.provisioning' + namespace: + - "{{data_stream.namespace}}" + - default + - target_dataset: azure.signinlogs + if: ctx.event?.dataset == 'azure.signinlogs' + namespace: + - "{{data_stream.namespace}}" + - default + - target_dataset: azure.springcloudlogs + if: ctx.event?.dataset == 'azure.springcloudlogs' + namespace: + - "{{data_stream.namespace}}" + - default diff --git a/packages/azure/docs/events.md b/packages/azure/docs/events.md new file mode 100644 index 00000000000..f3ebfd70006 --- /dev/null +++ b/packages/azure/docs/events.md @@ -0,0 +1,504 @@ +# Azure Logs Integration (v2 preview) + +The Azure Logs integration (v2 preview) collects logs from selected Azure services, such as Microsoft Entra ID (Sign-in, Audit, Identity Protection, and Provisioning logs), Azure Spring Apps, Azure Firewall, Microsoft Graph Activity, and several others. + +You can then visualize that data in Kibana, create alerts if something goes wrong, and reference data when troubleshooting an issue. + +For example, to detect possible brute force sign-in attacks, you can install the Azure Logs integration to send Azure sign-in logs to Elastic. Then, by setting up a new rule in the Elastic Observability Logs app, you can be alerted when the number of failed sign-in attempts exceeds a certain threshold. + +You may also want to plan your Azure capacity better. Send Azure Activity logs to Elastic to track and visualize when your virtual machines fail to start because they exceed the quota limit. + +## What's new in v2 preview? + +The Azure Logs integration (v2 preview) introduces a new architecture that allows you to forward logs from multiple Azure services to the same event hub. + +```text + ┌─────────────────┐ + │ activity logs │ + ┌─▶│ <> │ + │ └─────────────────┘ + │ +┌───────────────┐ ┌─────────────┐ ┌─────────────────┐ │ ┌─────────────────┐ +│ logs │ │ Elastic │ │ events (router) │ │ │ firewall logs │ +│ <> │──▶│ Agent │─▶│ <> │──┼─▶│ <> │ +└───────────────┘ └─────────────┘ └─────────────────┘ │ └─────────────────┘ + │ + │ ┌─────────────────┐ + │ │ signin logs │ + └─▶│ <> │ + └─────────────────┘ +``` + +The integration will automatically detect the log category and forward the logs to the appropriate data stream. When the integration v2 preview cannot find a matching data stream for a log category, it forwards the logs to the platform logs data stream. + +IMPORTANT: To use the v2 preview, you must turn off all the existing v1 integrations and turn on only the v2 preview integration. + +Under the hood, the v2 preview uses only one `azure-eventhub` input per event hub. The v2 preview avoids contention and inefficiencies from using multiple inputs with the same event hub, which is typical of the v1 architecture. With the v2 preview, you can still assign the agent policy to multiple Elastic Agents to scale out the logs processing. + +## Data streams + +The Azure Logs integration (v2 preview) collects logs. + +**Logs** help you keep a record of events that happen on your Azure account. +Log data streams collected by the Azure Logs integration include Activity, Platform, Microsoft Entra ID (Sign-in, Audit, Identity Protection, Provisioning), Microsoft Graph Activity, and Spring Apps logs. + +## Requirements + +You need Elasticsearch to store and search for your data and Kibana to visualize and manage it. +You can use our recommended hosted Elasticsearch Service on Elastic Cloud or self-manage the Elastic Stack on your hardware. + +Before using the Azure integration, you will need: + +* One or more **diagnostic settings** to export logs from Azure services to Event Hubs. +* One **event hub** to store in-flight logs exported by Azure services and make them available to Elastic Agent. +* One **Storage Account container** to store the event hub checkpointing information for each partition. + +### Diagnostic settings + +Azure diagnostic settings allow you to export metrics and logs from a **source** service (or resource) to one **destination** for analysis and long-term storage. + +```text + ┌────────────────────┐ ┌──────────────┐ ┌─────────────────┐ + │ Microsoft Entra ID │ │ Diagnostic │ │ Event Hub │ + │ <> │─────▶│ settings │────▶│ <> │ + └────────────────────┘ └──────────────┘ └─────────────────┘ +``` + +Examples of source services: + +* Azure Monitor +* Microsoft Entra ID +* Azure Firewall + +The diagnostic settings support several destination types. The Elastic Agent requires diagnostic settings configured with an event hub as the destination. + +### Event Hub + +[Azure Event Hubs](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-about) is a data streaming platform and event ingestion service that can receive and temporarily store millions of events. + +Elastic Agent with the Azure Logs integration will consume logs published in the Event Hubs service. + +```text + ┌────────────────┐ ┌────────────┐ + │ adlogs │ │ Elastic │ + │ <> │─────▶│ Agent │ + └────────────────┘ └────────────┘ +``` + +To learn more about Event Hubs, refer to [Features and terminology in Azure Event Hubs](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-features). + +### Storage account container + +The [Storage account](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview) is a versatile Azure service that allows you to store data in various storage types, including blobs, file shares, queues, tables, and disks. + +The Azure Logs integration requires a Storage Account container to work. The integration uses the Storage Account container for checkpointing; it stores data about the Consumer Group (state, position, or offset) and shares it among the Elastic Agents. Sharing such information allows multiple Elastic Agents assigned to the same agent policy to work together, enabling horizontal scaling of the logs processing when required. + +```text + ┌────────────────┐ ┌────────────┐ + │ adlogs │ logs │ Elastic │ + │ <> │────────────────────▶│ Agent │ + └────────────────┘ └────────────┘ + │ + consumer group info │ + ┌────────────────┐ (state, position, or │ + │ azurelogs │ offset) │ + │ <> │◀───────────────────────────┘ + └────────────────┘ +``` + +The Elastic Agent automatically creates one container for each enabled integration. In the container, the Agent will create one blob for each existing partition on the event hub. + +For example, if you enable one integration to fetch data from an event hub with four partitions, the Agent will create the following: + +* One Storage Account container. +* Four blobs in that container. + +The information stored in the blobs is small (usually < 300 bytes per blob) and accessed relatively frequently. Elastic recommends using the Hot storage tier. + +You need to keep the Storage Account container as long as you need to run the integration with the Elastic Agent. If you delete a storage account container, the Elastic Agent will stop working and create a new one the next time it starts. By deleting a storage account container, the Elastic Agent will lose track of the last message processed and start processing messages from the beginning of the event hub retention period. + +## Setup + +With the Azure Logs integration (v2 preview), you can forward logs from multiple Azure services to the same event hub. The integration will automatically detect the log category and forward the logs to the appropriate data stream. + +```text + ┌─────────────────┐ + │ activity logs │ + ┌─▶│ <> │ + │ └─────────────────┘ + │ +┌───────────────┐ ┌─────────────┐ ┌─────────────────┐ │ ┌─────────────────┐ +│ logs │ │ Elastic │ │ events (router) │ │ │ firewall logs │ +│ <> │──▶│ Agent │─▶│ <> │──┼─▶│ <> │ +└───────────────┘ └─────────────┘ └─────────────────┘ │ └─────────────────┘ + │ + │ ┌─────────────────┐ + │ │ signin logs │ + └─▶│ <> │ + └─────────────────┘ +``` + +Before adding the integration, you must complete the following tasks. + +### Create an Event Hub + +The event hub receives the logs exported from the Azure service and makes them available for the Elastic Agent to read. + +Here's a high-level overview of the required steps: + +* Create a resource group, or select an existing one. +* Create an Event Hubs namespace. +* Create an event hub. + +For a detailed step-by-step guide, check the quickstart [Create an event hub using Azure portal](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-create). + +Take note of the event hub **Name**, which you will use later when specifying an **eventhub** in the integration settings. + +#### Event Hubs Namespace vs Event Hub + +In the integration settings, you should use the event hub name (not the Event Hubs namespace name) as the value for the **event hub ** option. + +If you are new to Event Hubs, think of the Event Hubs namespace as the cluster and the event hub as the topic. You will typically have one cluster and multiple topics. + +If you are familiar with Kafka, here's a conceptual mapping between the two: + +| Kafka Concept | Event Hub Concept | +|----------------|-------------------| +| Cluster | Namespace | +| Topic | An event hub | +| Partition | Partition | +| Consumer Group | Consumer Group | +| Offset | Offset | + +#### How many partitions? + +The number of partitions is essential to balance the event hub cost and performance. + +Here are a few examples with one or multiple agents, with recommendations on picking the correct number of partitions for your use case. + +##### Single Agent + +With a single Agent deployment, increasing the number of partitions on the event hub is the primary driver in scale-up performances. The Agent creates one worker for each partition. + +```text +┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ + +│ │ │ │ + +│ ┌─────────────────┐ │ │ ┌─────────────────┐ │ + │ partition 0 │◀───────────│ consumer │ +│ └─────────────────┘ │ │ └─────────────────┘ │ + ┌─────────────────┐ ┌─────────────────┐ +│ │ partition 1 │◀──┼────┼───│ consumer │ │ + └─────────────────┘ └─────────────────┘ +│ ┌─────────────────┐ │ │ ┌─────────────────┐ │ + │ partition 2 │◀────────── │ consumer │ +│ └─────────────────┘ │ │ └─────────────────┘ │ + ┌─────────────────┐ ┌─────────────────┐ +│ │ partition 3 │◀──┼────┼───│ consumer │ │ + └─────────────────┘ └─────────────────┘ +│ │ │ │ + +│ │ │ │ + +└ Event Hub ─ ─ ─ ─ ─ ─ ─ ┘ └ Agent ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ +``` + +##### Two or more Agents + +With more than one Agent, setting the number of partitions is crucial. The agents share the existing partitions to scale out performance and improve availability. + +The number of partitions must be at least the number of agents. + +```text +┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ + +│ │ │ ┌─────────────────┐ │ + ┌──────│ consumer │ +│ ┌─────────────────┐ │ │ │ └─────────────────┘ │ + │ partition 0 │◀────┘ ┌─────────────────┐ +│ └─────────────────┘ │ ┌──┼───│ consumer │ │ + ┌─────────────────┐ │ └─────────────────┘ +│ │ partition 1 │◀──┼─┘ │ │ + └─────────────────┘ ─Agent─ ─ ─ ─ ─ ─ ─ ─ ─ ─ +│ ┌─────────────────┐ │ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ + │ partition 2 │◀────┐ +│ └─────────────────┘ │ │ │ ┌─────────────────┐ │ + ┌─────────────────┐ └─────│ consumer │ +│ │ partition 3 │◀──┼─┐ │ └─────────────────┘ │ + └─────────────────┘ │ ┌─────────────────┐ +│ │ └──┼──│ consumer │ │ + └─────────────────┘ +│ │ │ │ + +└ Event Hub ─ ─ ─ ─ ─ ─ ─ ┘ └ Agent ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ +``` + +##### Recommendations + +Create an event hub with at least two partitions. Two partitions allow low-volume deployment to support high availability with two agents. Consider creating four partitions or more to handle medium-volume deployments with availability. + +To learn more about event hub partitions, read an in-depth guide from Microsoft at [Quickstart: Create an event hub using Azure portal](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-create). + +To learn more about event hub partition from the performance perspective, check the scalability-focused document at [Event Hubs scalability](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-scalability#partitions). + +#### How many Event Hubs? + +With the Azure Logs integration (v2 preview), Elastic strongly recommends creating one event hub and using it for all Azure services. + +For example, if you plan to collect Microsoft Entra ID and Azure Firewall logs, create one event hub and use it for both services. + +Here's a high-level diagram of the solution: + +```text +┌────────────────┐ ┌───────────────┐ +│ MS Entra ID │ │ Diagnostic │ +│ <> │──▶│ Settings │─┐ +└────────────────┘ └───────────────┘ │ + │ ┌───────────────┐ ┌─────────────┐ + │ │ logs │ │ Elastic │ + ├─▶│ <> │──▶│ Agent │ + │ └───────────────┘ └─────────────┘ +┌────────────────┐ ┌───────────────┐ │ +│ Azure Firewall │ │ Diagnostic │ │ +│ <> │──▶│ Settings │─┘ +└────────────────┘ └───────────────┘ +``` + +The Azure Logs integration (v2 preview) will automatically detect the log category and forward the logs to the appropriate data stream. + +#### Consumer Group + +Like all other event hub clients, Elastic Agent needs a consumer group name to access the event hub. + +A Consumer Group is an entire event hub's view (state, position, or offset). Consumer groups enable multiple agents to have a separate view of the event stream and to read the logs independently at their own pace and with their offsets. + +Consumer groups allow multiple Elastic Agents assigned to the same agent policy to work together; this enables horizontal scaling of the logs processing when required. + +In most cases, you can use the default consumer group named `$Default`. If `$Default` is already used by other applications, you can create a consumer group dedicated to the Azure Logs integration. + +#### Connection string + +The Elastic Agent requires a connection string to access the event hub and fetch the exported logs. The connection string contains details about the event hub used and the credentials required to access it. + +To get the connection string for your Event Hubs namespace: + +1. Visit the **Event Hubs namespace** you created in a previous step. +1. Select **Settings** > **Shared access policies**. + +Create a new Shared Access Policy (SAS): + +1. Select **Add** to open the creation panel. +1. Add a **Policy name** (for example, "ElasticAgent"). +1. Select the **Listen** claim. +1. Select **Create**. + +When the SAS Policy is ready, select it to display the information panel. + +Take note of the **Connection string–primary key**, which you will use later when specifying a **connection_string** in the integration settings. + +### Create a diagnostic settings + +The diagnostic settings export the logs from Azure services to a destination and in order to use Azure Logs integration, it must be an event hub. + +To create a diagnostic settings to export logs: + +1. Locate the diagnostic settings for the service (for example, Microsoft Entra ID). +2. Select diagnostic settings in the **Monitoring** section of the service. Note that different services may place the diagnostic settings in various positions. +3. Select **Add diagnostic settings**. + +In the diagnostic settings page you must select the source **log categories** you want to export and then select their **destination**. + +#### Select log categories + +Each Azure service exports a well-defined list of log categories. Check the individual integration doc to learn which log categories the integration supports. + +#### Select the destination + +Select the **subscription** and the **Event Hubs namespace** you previously created. Select the event hub dedicated to this integration. + +```text + ┌───────────────┐ ┌──────────────┐ ┌───────────────┐ ┌───────────┐ + │ MS Entra ID │ │ Diagnostic │ │ adlogs │ │ Elastic │ + │ <> ├──▶│ Settings │──▶│ <> │─────▶│ Agent │ + └───────────────┘ └──────────────┘ └───────────────┘ └───────────┘ +``` + +### Create a Storage Account container + +The Elastic Agent stores the event hub checkpoint information in a storage account container. Storing checkpoint information in a container allows agents to share message processing and resume from the last processed message after a restart. + +**Note**: Use the Storage Account as a checkpoint store only. + +To create the storage account: + +1. Sign in to the [Azure Portal](https://portal.azure.com/) and create your storage account. +2. While configuring your project details, make sure you select the following recommended default settings: + * Hierarchical namespace: disabled + * Minimum TLS version: Version 1.2 + * Access tier: Hot + * Enable soft delete for blobs: disabled + * Enable soft delete for containers: disabled + +3. When the new storage account is ready, you need to take note of the storage account name and the Storage Account access keys, as you will use them later to authenticate your Elastic application’s requests to this storage account. + +This is the final diagram of the setup for collecting Activity logs from the Azure Monitor service. + +```text + ┌───────────────┐ ┌──────────────┐ ┌────────────────┐ ┌───────────┐ + │ MS Entra ID │ │ Diagnostic │ │ adlogs │ logs │ Elastic │ + │ <> ├──▶│ Settings │──▶│ <> │────────▶│ Agent │ + └───────────────┘ └──────────────┘ └────────────────┘ └───────────┘ + │ + ┌──────────────┐ consumer group info │ + │ azurelogs │ (state, position, or │ + │<> │◀───────────────offset)──────────────┘ + └──────────────┘ +``` + +#### How many Storage Accounts? + +The Elastic Agent can use a single Storage Account to store the checkpoint information for multiple integrations. + +**CRITICAL**: make sure to use a different **storage_account_container** for each integration. The Elastic Agent uses the **integration name** and the **event hub name** to uniquely identify the container that holds the blobs with the checkpoint information. + +```text +┌─────────────────────────────────┐ ┌──────────────────────────────────────────┐ +│ │ │ │ +│ ┌─────────────────────┐ │ │ ┌────────────────────────────────────┐ │ +│ │ azure-eventhub │ │ │ │ filebeat-activitylogs-eventhub-1 │ │ +│ │ <> │──────┼──────┼─▶│ <> │ │ +│ └─────────────────────┘ │ │ └────────────────────────────────────┘ │ +│ ┌─────────────────────┐ │ │ ┌────────────────────────────────────┐ │ +│ │ azure-eventhub │ │ │ │ filebeat-signinlogs-eventhub-2 │ │ +│ │ <> │──────┼──────┼─▶│ <> │ │ +│ └─────────────────────┘ │ │ └────────────────────────────────────┘ │ +│ ┌─────────────────────┐ │ │ ┌────────────────────────────────────┐ │ +│ │ azure-eventhub │ │ │ │ filebeat-auditlogs-eventhub-3 │ │ +│ │ <> │──────┼──────┼─▶│ <> │ │ +│ └─────────────────────┘ │ │ └────────────────────────────────────┘ │ +│ │ │ │ +└─Elastic Agent───────────────────┘ └─Storage Account──────────────────────────┘ +``` + +### Running the integration behind a firewall + +When you run the Elastic Agent behind a firewall, you must allow traffic on ports `5671` and `5672` for the event hub and port `443` for the Storage Account container to ensure proper communication with the necessary components. + +```text +┌────────────────────────────────┐ ┌───────────────────┐ ┌───────────────────┐ +│ │ │ │ │ │ +│ ┌────────────┐ ┌───────────┐ │ │ ┌──────────────┐ │ │ ┌───────────────┐ │ +│ │ diagnostic │ │ event hub │ │ │ │azure-eventhub│ │ │ │ activity logs │ │ +│ │ setting │──▶│ │◀┼AMQP─│ <> │─┼──┼▶│<>│ │ +│ └────────────┘ └───────────┘ │ │ └──────────────┘ │ │ └───────────────┘ │ +│ │ │ │ │ │ │ +│ │ │ │ │ │ │ +│ │ │ │ │ │ │ +│ ┌─────────────┬─────HTTPS─┼──────────┘ │ │ │ +│ ┌───────┼─────────────┼──────┐ │ │ │ │ │ +│ │ │ │ │ │ │ │ │ │ +│ │ ▼ ▼ │ │ └─Agent─────────────┘ └─Elastic Cloud─────┘ +│ │ ┌──────────┐ ┌──────────┐ │ │ +│ │ │ 0 │ │ 1 │ │ │ +│ │ │ <> │ │ <> │ │ │ +│ │ └──────────┘ └──────────┘ │ │ +│ │ │ │ +│ │ │ │ +│ └─Storage Account Container──┘ │ +│ │ +│ │ +└─Azure──────────────────────────┘ +``` + +#### Event Hub + +Port `5671` and `5672` are commonly used for secure communication with the event hub. These ports are used to receive events. The Elastic Agent can establish a secure connection with the event hub by allowing traffic on these ports. + +For more information, check the following documents: + +* [What ports do I need to open on the firewall?](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-faq#what-ports-do-i-need-to-open-on-the-firewall) from the [Event Hubs frequently asked questions](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-faq#what-ports-do-i-need-to-open-on-the-firewall). +* [AMQP outbound port requirements](https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-protocol-guide#amqp-outbound-port-requirements) + +#### Storage Account container + +The Elastic Agent uses port `443` for secure communication with the Storage Account container. By allowing traffic on port 443, the Elastic Agent can securely access and interact with the Storage Account container, essential for storing and retrieving checkpoint data for each event hub partition. + +#### DNS + +Optionally, you can restrict the traffic to the following domain names: + +```text +*.servicebus.windows.net +*.blob.core.windows.net +*.cloudapp.net +``` + +## Settings + +Use the following settings to configure the Azure Logs integration when you add it to Fleet. + +`eventhub` : +_string_ +A fully managed, real-time data ingestion service. Elastic recommends using only letters, numbers, and the hyphen (-) character for event hub names to maximize compatibility. You can use existing event hubs having underscores (_) in the event hub name; in this case, the integration will replace underscores with hyphens (-) when it uses the event hub name to create dependent Azure resources behind the scenes (e.g., the storage account container to store event hub consumer offsets). Elastic also recommends using a separate event hub for each log type as the field mappings of each log type differ. +Default value `insights-operational-logs`. + +`consumer_group` : +_string_ +Enable the publish/subscribe mechanism of Event Hubs with consumer groups. A consumer group is a view (state, position, or offset) of an entire event hub. Consumer groups enable multiple consuming applications to each have a separate view of the event stream, and to read the stream independently at their own pace and with their own offsets. +Default value: `$Default` + +`connection_string` : +_string_ + +The connection string is required to communicate with Event Hubs. See [Get an Event Hubs connection string](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string) for more information. + +A Blob Storage account is required to store/retrieve/update the offset or state of the event hub messages. This allows the integration to start back up when it stopped processing messages. + +`storage_account` : +_string_ +The name of the storage account that the state/offsets will be stored and updated. + +`storage_account_key` : +_string_ +The storage account key. Used to authorize access to data in your storage account. + +`storage_account_container` : +_string_ +The storage account container where the integration stores the checkpoint data for the consumer group. It is an advanced option to use with extreme care. You MUST use a dedicated storage account container for each Azure log type (activity, sign-in, audit logs, and others). DO NOT REUSE the same container name for more than one Azure log type. See [Container Names](https://docs.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata#container-names) for details on naming rules from Microsoft. The integration generates a default container name if not specified. + +`resource_manager_endpoint` : +_string_ +Optional. By default, the integration uses the Azure public environment. To override this and use a different Azure environment, users can provide a specific resource manager endpoint + +Examples: + +* Azure ChinaCloud: `https://management.chinacloudapi.cn/` +* Azure GermanCloud: `https://management.microsoftazure.de/` +* Azure PublicCloud: `https://management.azure.com/` +* Azure USGovernmentCloud: `https://management.usgovcloudapi.net/` + +This setting can also be used to define your own endpoints, like for hybrid cloud models. + +## Handling Malformed JSON in Azure Logs + +Azure services have been observed occasionally sending [malformed JSON](https://learn.microsoft.com/en-us/answers/questions/1001797/invalid-json-logs-produced-for-function-apps) documents. These logs can disrupt the expected JSON formatting and lead to parsing issues during processing. + +To address this issue, the advanced settings section of each data stream offers two sanitization options: + +* Sanitizes New Lines: removes new lines in logs. +* Sanitizes Single Quotes: replace single quotes with double quotes in logs, excluding single quotes occurring within double quotes. + +Malformed logs can be identified by: + +* The presence of a records array in the message field indicates a failure to unmarshal the byte slice. +* An `error.message` field contains the "Received invalid JSON from the Azure Cloud platform. Unable to parse the source log message" text. + +Known data streams that might produce malformed logs: + +* Platform Logs +* Spring Apps Logs + +## Reference + +Visit the page for each individual Azure Logs integration to see details about exported fields and sample events. diff --git a/packages/azure/manifest.yml b/packages/azure/manifest.yml index aae5df25d6c..7bc14587b86 100644 --- a/packages/azure/manifest.yml +++ b/packages/azure/manifest.yml @@ -1,6 +1,6 @@ name: azure title: Azure Logs -version: 1.19.4 +version: "1.20.0" description: This Elastic integration collects logs from Azure type: integration icons: @@ -8,7 +8,7 @@ icons: title: logo azure size: 32x32 type: image/svg+xml -format_version: "3.0.2" +format_version: "3.3.0" categories: - azure - observability @@ -71,6 +71,28 @@ vars: required: false show_user: false policy_templates: + - name: events + title: Azure Logs (v2 preview) + description: Azure Logs (v2 preview) integration + data_streams: + - events + categories: + - stream_processing + inputs: + - type: "azure-eventhub" + title: "Collect all Azure Logs (v2 preview)" + description: "Collecting log events from Azure Event Hub (input: azure-eventhub)" + input_group: logs + icons: + - src: /img/eventhub.png + title: logo azure + size: 32x32 + type: image/svg+xml + screenshots: + - src: /img/filebeat-azure-overview.png + title: filebeat azure overview + size: 5002x2666 + type: image/png - name: eventhub title: Azure Event Hub Input description: Azure Event Hub input integration @@ -80,8 +102,8 @@ policy_templates: - stream_processing inputs: - type: "azure-eventhub" - title: "Collect events from Event Hub" - description: "Collecting events from Azure Event Hub inputs (input: azure-eventhub)" + title: "Collect raw events (v1)" + description: "Collecting raw events from Azure Event Hub inputs (input: azure-eventhub)" input_group: logs icons: - src: /img/eventhub.png @@ -105,7 +127,7 @@ policy_templates: - security inputs: - type: "azure-eventhub" - title: "Collect Microsoft Entra ID logs from Event Hub" + title: "Collect Microsoft Entra ID logs (v1)" description: "Collecting Microsoft Entra ID logs as audit logs and signin logs from Azure instances (input: azure-eventhub)" input_group: logs icons: @@ -125,7 +147,7 @@ policy_templates: - platformlogs inputs: - type: "azure-eventhub" - title: "Collect Azure platform logs from Event Hub" + title: "Collect Azure platform logs (v1)" description: "Collecting platform logs from Azure instances (input: azure-eventhub)" input_group: logs icons: @@ -145,7 +167,7 @@ policy_templates: - activitylogs inputs: - type: "azure-eventhub" - title: "Collect Azure Activity Logs from Event Hub" + title: "Collect Azure Activity Logs (v1)" description: "Collecting activity logs from Azure instances (input: azure-eventhub)" input_group: logs icons: @@ -167,7 +189,7 @@ policy_templates: - security inputs: - type: "azure-eventhub" - title: "Collect Microsoft Graph Activity Logs from Event Hub" + title: "Collect Microsoft Graph Activity Logs (v1)" description: "Collecting graph activity logs from Azure instances (input: azure-eventhub)" input_group: logs icons: @@ -187,7 +209,7 @@ policy_templates: - springcloudlogs inputs: - type: "azure-eventhub" - title: "Collect Azure Spring Apps logs from Event Hub" + title: "Collect Azure Spring Apps logs (v1)" description: "Collecting Spring Apps logs from Azure instances (input: azure-eventhub)" input_group: logs icons: @@ -210,7 +232,7 @@ policy_templates: - firewall_security inputs: - type: "azure-eventhub" - title: "Collect Azure firewall logs from Event Hub" + title: "Collect Azure firewall logs (v1)" description: "Collecting firewall logs from Azure (input: azure-eventhub)" input_group: logs icons: @@ -236,7 +258,7 @@ policy_templates: - security inputs: - type: "azure-eventhub" - title: "Collect Azure Application Gateway logs from Event Hub" + title: "Collect Azure Application Gateway logs (v1)" description: "Collecting Application Gateway logs from Azure (input: azure-eventhub)" input_group: logs icons: