> **Page index:** https://km.oslo.systems/llms.txt lists every page on this site, with a link to its Markdown version.

# Customize OTel configuration

Customize the OTel configuration to control which telemetry your application sends to Datadog.

## Things to consider

> [!WARNING]
>
> **Custom metric pricing**
>
> Custom metrics can help you understand your business and systems, but they can be expensive. Datadog and most observability vendors treat each unique combination of metric attribute values as a separate metric.
>
> Be conscious of the number of custom metrics you emit. In particular, avoid metric attributes that take on a high number of values (also called high-dimensionality), such as user IDs, IP addresses, and timestamps.
>
> If removing certain metrics or metric attributes at the application layer is difficult, you can have the OTel collector filter them out by customizing the OTel configuration.

## Before you begin

Make sure you have:

- Completed the [getting started guide to set up basic Datadog integration](<https://km.oslo.systems/initial-setup-of-your-aws-environment/set-up-datadog/index.md>).
- An application deployed using the `app` template with telemetry-collection enabled.

## Filter out specific metrics

stacks/dev/app-example/config\_override.tf

```
locals {
  otel_metric_config = {
    ignored_metrics = [
      # NOTE: We always want to remove the metrics below
      "^http\\.server\\.request\\.duration$",
      "^http\\.client\\.request\\.duration$",
      "^otlp\\.exporter\\.exported$",
      "^otlp\\.exporter\\.seen$",
      "^queueSize$",
      "^processedLogs$",
      "^processedSpans$",
      # TODO: Add the name of the metric you want to remove here
    ]
  }
}
```

## Remove metric attributes

stacks/dev/app-example/config\_override.tf

```
locals {
  # We add a new OTel processor to the OTel configuration
  otel_additional_processors = {
    "transform/remove_metric_attributes" = {
      metric_statements = [
        # Remove an attribute from all metrics
        "delete_key(datapoint.attributes, \"example.attribute\")",
        # Remove attribute only from specific metric
        "delete_key(datapoint.attributes, \"example.attribute\") where metric.name == \"system.memory.usage\""
      ]
    }
  }

  # We add our new processor to the OTel pipeline that contains metrics
  otel_pipeline_processors = {
    "metrics" = {
      processors = ["transform/remove_metric_attributes"]
    }
  }
}
```

## Only keep traces from a specific endpoint

stacks/dev/app-example/config\_override.tf

```
locals {
  # Configure tail sampling to only keep traces from a specific endpoint
  otel_tail_sampling_config = {
    additional_policies = [
      {
        name = "drop-everything-but-one-route"
        type = "drop"
        drop = {
          drop_sub_policy = [
            {
              name = "inverted-match"
              type = "string_attribute"
              string_attribute = {
                key                    = "http.route"
                values                 = ["^/my-route$"]
                enabled_regex_matching = true
                invert_match           = true
              }
            }
          ]
        }
      }
    ]
  }
}
```

## Bring your own OTel Collector configuration

> [!WARNING]
>
> When you bring your own OTel configuration, you'll lose useful behavior and telemetry tags that the Golden Path provides out-of-the-box. Consider using the `otel-config-generator` Terraform module to extend the default configuration rather than replacing it, or ask Utviklerflyt to support your use case.

stacks/dev/app-example/config\_override.tf

```
locals {
  otel_config = yamlencode(file("my-config.yml"))
}
```