How can i check AccountingService's Metrics?

[eee269]

I'm deploying the OpenTelemetry demo on a Kubernetes cluster and encountering an issue. Here's my setup:

Resource Configuration

• A Namespace: OpenTelemetry demo, OpenTelemetry collector (deployment, daemonset).
• B Namespace: OpenTelemetry collector (deployment).
• Default Namespace: OpenTelemetry operator.

Workflow

1. Using auto-instrumentation to collect performance metrics.
2. In the A Namespace, the OpenTelemetry demo sends data to an OpenTelemetry collector (daemonset), which forwards it to the OpenTelemetry collector in the B Namespace.
3. In the B Namespace, the data is sent to ClickHouse and Kafka.

Question

How can I verify metrics for the accounting service in ClickHouse?
Currently, I can't find any metrics using ServiceName=accountingservice. However, I can find metrics for adservice and recommendationservice.
Do I need to configure additional settings for auto-instrumentation?
Could you please guide me on what could be missing or misconfigured?

---

Current Auto-Instrumentation Setup

1. OpenTelemetry demo, and collectors in both A and B Namespaces are already deployed.
2. OpenTelemetry operator is deployed in the default namespace using Helm scripts.
3. An instrumentation file is applied in the A Namespace, with the endpoint pointing to the OpenTelemetry collector in the B Namespace.

# auto-instrumentation.yaml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: demo-instrumentation
  namespace: A
spec:
  exporter:
    endpoint: http://192.168.xxx.xxx:4317
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1"
  dotnet:
    env:
      - name: OTEL_DOTNET_AUTO_TRACES_GRPCNETCLIENT_INSTRUMENTATION_ENABLED
        value: 'true'
      - name: OTEL_DOTNET_AUTO_METRICS_PROCESS_INSTRUMENTATION_ENABLED
        value: 'true'
  java:
    env:
      - name: OTEL_INSTRUMENTATION_KAFKA_ENABLED
        value: 'true'
      - name: OTEL_INSTRUMENTATION_REDISCALA_ENABLED
        value: 'true'
  nodejs:
    env:
      - name: OTEL_NODE_ENABLED_INSTRUMENTATIONS
        value: http,nestjs-core
      - name: OTEL_NODE_DISABLED_INSTRUMENTATIONS
        value: fs,grpc
  python:
    env:
      - name: OTEL_LOGS_EXPORTER
        value: otlp_proto_http
      - name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
        value: 'true'

[julianocosta89]

Hello @eee269 👋🏽

recomedationservice is a Python service and adservice is a Java service.

Are you getting any metrics from the cartservice?
cartservice is also .NET, so it would be a better service to compare.

I'm not experienced with the operator and that is not a scenario we support in the demo currently, but I may be able to help you out.

Just one question, have you uninstrumented the services?

[eee269]

Thanks @julianocosta89 !!🙌

I already know that accountingservice and cartservice are .NET service.
And I can't found metrics from the cartservice.
But leggacy service of the Manual Instrumentation has metrics like this.

1. http
http.client.connection.duration (histogram)
http.client.request.duration (histogram)
http.client.request.time_in_queue (histogram)
http.server.request.duration (histogram)
http.client.active_requests (sum)
http.client.open_connections (sum)
http.server.active_requests (sum)
2. process_memory
process.memory.usage (sum)
process.memory.virtual (sum)
3. process_cpu
process.cpu.count (sum)
process.cpu.time (sum)
4. process_thread
process.thread.count (sum)
5. dotnet_process_gc
process.runtime.dotnet.gc.allocations.size (sum)
process.runtime.dotnet.gc.collections.count (sum)
process.runtime.dotnet.gc.duration (sum)
process.runtime.dotnet.gc.objects.size (sum)
6. dotnet_process_jit
process.runtime.dotnet.jit.compilation_time (sum)
[process.runtime.dotnet.jit.il](http://process.runtime.dotnet.jit.il/)_compiled.size (sum)
process.runtime.dotnet.jit.methods_compiled.count (sum)
7. dotnet_process_thread
process.runtime.dotnet.thread_pool.completed_items.count (sum)
process.runtime.dotnet.thread_pool.queue.length (sum)
process.runtime.dotnet.thread_pool.threads.count (sum)
8. dotnet_kestrel
kestrel.connection.duration (histogram)
kestrel.active_connections (sum)
kestrel.queued_connections (sum)
9. dotnet_routing
aspnetcore.routing.match_attempts (sum)
10. dotnet_misc
process.runtime.dotnet.monitor.lock_contention.count (sum)
process.runtime.dotnet.timer.count (sum)
process.runtime.dotnet.assemblies.count (sum)
process.runtime.dotnet.exceptions.count (sum)

---

have you uninstrumented the services?

No, Services are instrumented now. I deleted and reapplied instrumentation service that use auto-instrumentation.yaml for 1-2 times

[julianocosta89]

No, Services are instrumented now. I deleted and reapplied instrumentation service that use auto-instrumentation.yaml for 1-2 times

I don't understand what you mean here.
accountingservice and cartservice are already instrumented in the demo as of now.
The operator would try to add instrumentation to an already instrumented service, which would probably cause some conflicts