Observability with OpenTelemetry

[mcarlett]

Since OpenTelemetry is becoming a standard, it would be nice to have some infra ready to manage data (metrics/traces/logs) using OpenTelemetry collector.
To achieve this goal we need to install some of infrastructural components besides OpenTelemetry and Observability operators such as:

• Grafana Tempo as storage for traces
• Grafana Loki as storage for logs
• MinIO as cross storage

[ammachado]

Some suggestions:

• https://github.com/grafana/docker-otel-lgtm (Official Loki+Grafana+Tempo+Prometheus image, ephemeral storage)
• https://github.com/nunchistudio/docker-opentelemetry (local dev stack for OpenTelemetry)

[mcarlett]

Some suggestions:

• https://github.com/grafana/docker-otel-lgtm (Official Loki+Grafana+Tempo+Prometheus image, ephemeral storage)
• https://github.com/nunchistudio/docker-opentelemetry (local dev stack for OpenTelemetry)

thanks @ammachado ! Those contain useful configurations, where it possible, I would prefer to use operators

[davgordo]

Thank you for getting this conversation started @mcarlett I think it's a really important piece of the platform that hasn't had any attention on it yet.

I suggest that we may benefit from breaking it down into smaller pieces for individuals to contribute/review the parts more easily. We could use this a parent issue.

Suggested sub-issues:

• Enable user-workload monitoring on the platform
  https://docs.openshift.com/container-platform/4.17/observability/monitoring/enabling-monitoring-for-user-defined-projects.html
• Deploy a Grafana instance for custom charts
  https://github.com/redhat-cop/gitops-catalog/tree/main/grafana-operator/overlays/user-app
• Deploy a Grafana Tempo instance (with Minio storage)
  https://docs.openshift.com/container-platform/4.17/observability/distr_tracing/distr_tracing_tempo/distr-tracing-tempo-installing.html
• Configure log forwarding to Loki with LokiStack storage
  https://docs.openshift.com/container-platform/4.17/observability/logging/logging-6.0/log6x-clf.html
  https://docs.openshift.com/container-platform/4.17/observability/logging/logging-6.0/log6x-loki.html

For the Grafana instance, the Grafana Labs operator is my recommendation. It is not Red Hat supported, but it was contributed by the integrately team, so I consider it close enough to the Red Hat community. It is the best way to host custom charts. We are not supposed to customize the charts that are built-into OpenShift if we want to stick with a Red Hat supported configuration for the platform.

What to you guys think about the breakdown of sub issues?

[mcarlett]

thank you @davgordo I think it is a good idea to create sub-tasks, I was wondering if, besides Grafana dashboard we can use also Cluster Observability Operator, even if it is in tech preview, but it handles UI plugins integrated in the OCP console to visualize traces and logs

[davgordo]

That's a great idea @mcarlett let's see what we can do with that. And for now, we can prioritize the OpenShift-native charting platform unless/until we find limitations that get in our way. Thanks for the note, good find.