AWS::Logs::MetricFilter - Dimension support

[brentcetinich]

I should be able to hard code a dimension (or use Fn::Sub) to specify a Dimension value, rather than always pull it from a log line.

Use cases:

• Filtering lambda logs that don't contain the function name.
• Attributing logs to deployment environments (prod, qa, etc)

This could be easily achieved by allowing non-selectors as dimension values (strings that don't start with $)

Currently if you try do do this you get:

        metricName: `MemoryUsed`,
        metricValue: "$some_value",
         dimensions: {
           FunctionName: `${node.ref}`
         },

Resource handler returned message "invalid request provided: AWS::Logs::MetricFilter. Invalid metric transformation: dimension values must be valid selector.

So instead I need to inject the lambda name into the metric name - This makes looking for the metric and reporting on it a miserable experience, but to get it to works I have to do this :

        metricName: `MemoryUsed-${node.ref}`,
        metricValue: "$some_value",

See https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-cloudwatch-logs-announces-dimension-support-for-metric-filters/

[gordonpn]

Hi Brent, thank you for raising this issue. We understand the pain of having to use dynamic dimension values, and I have raised the same issue internally. I will keep this GitHub issue updated if there is any progress on our side. Thanks again!

[r-heimann]

Any news? Would love to have this.

[josiahcoad]

Would love this :) same use-case as this guy. I am creating a custom metric/alarm for memory usage on my lambdas and I don't have a great way of identifying the different metrics (except for putting the name of the function in the metric name which is not ideal)

[jackson-theisen]

Another great use case for this would be for Lambda timeout metrics. When timeouts occur, a "platform log event" is emitted: 2024-04-18T19:29:10.413Z 21c2cf65-ce53-40ff-920f-720f2c52ae41 Task timed out after 3.02 seconds.

It's easy enough to create a metric filter for Task timed out logs, but since the function name is not included in the log (for both Plaintext and JSON formats), there is no good way to correlate a timeout to a function in CloudWatch Metrics unless you create a unique namespace for every function (not scalable). If we could just hard-code the function name as a dimension, the problem would be solved.

[sean-r-williams]

@gordonpn Is this still on your team's radar? In the various GitHub issues related to this feature ask, I haven't seen any movement in the last year or so.

We have a Lambda function (with associated CloudWatch logs) that we are trying to monitor in DataDog. We use LMFs for extracting key data out of lambda runtime startup, and we don't have control over the runtime's log format. We need to imbue these metrics with additional [static-value] dimensions so we can track service ownership within Datadog.

The inability to add dimensions is hurting our observability story for these Lambdas. We would like to avoid having to replace our CloudWatch logs/metrics with native Datadog functionality if possible.

[moosius]

It looks like the recently announced log group transformation ability does provide a way out, albeit a clunky one. You can use a transformer to add static values, then reference them in the metric filter. I was able to create a timeout metric using this approach. The relevant Cloudformation snippet is shown below. (I actually use CDK and edited the output to try to make it more readable, so apologies if I mistyped something but hopefully the idea comes through.)

  TestFunctionTransformer:
    Type: AWS::Logs::Transformer
    Properties:
      LogGroupIdentifier: /aws/lambda/lambda-transform-test-function
      TransformerConfig:
        - ParseJSON: {}
        - AddKeys:
            Entries:
              - Key: functionName
                Value: lambda-transform-test-function
                OverwriteIfExists: false
  TestFunctionTimeoutMetricFilter:
    Type: AWS::Logs::MetricFilter
    Properties:
      FilterPattern: '{ ($.type = "platform.report") && ($.record.status = "timeout") }'
      LogGroupName: /aws/lambda/lambda-transform-test-function
      MetricTransformations:
        - Dimensions:
            - Key: Function
              Value: $.functionName
          MetricName: LambdaTimeout
          MetricNamespace: Test
          MetricValue: "1"
          Unit: Count
      ApplyOnTransformedLogs: "true"

That will create a "LambdaTimeout" metric in the "Test" namespace. (The log group is set to use json logs, which is why the FilterPattern uses JSON selector.) The metric filter references the "$.functionName" injected by the "AddKeys" transformer processor.

[r-heimann]

Thank you for your example. Since the Log Group Transformations has been implemented maybe the CloudWatch Logs team finally has some time to work on their technical debt, like this issue.