Grafana agent API endpoints not working

[JCastrejonE]

I'm running a grafana agent inside docker, with the following agent.yaml

metrics:
  global:
    scrape_interval: 60s
  configs:
  - name: hosted-prometheus
    scrape_configs:
      - job_name: ${GRAFANA_JOB_NAME}
        static_configs:
        - targets: [${COLLECTOR_HOST}]
    remote_write:
      - url: https://${GRAFANA_PROM_HOST}/api/prom/push
        basic_auth:
          username: ${GRAFANA_PROM_USER}
          password: ${GRAFANA_PASS}

Inside my docker compose:

agent:
    image: grafana/agent:latest
    volumes:
      - ./agent/config:/etc/agent-config
    environment:
      GRAFANA_JOB_NAME: "local/${NETWORK}"
    env_file:
      - .env
    entrypoint:
      - /bin/agent
      - -config.file=/etc/agent-config/agent.yaml
      - -config.expand-env=true
    ports:
      - 12345:12345

However, when the agent starts I don't get a response from http://localhost:12345/-/healthy or any of the API endpoints. I can confirm that the port is being used because the error is not from connection refused, it is ERR_EMPTY_RESPONSE. I need to use the healthy or the ready endpoints because this is failing healthchecks when deploying to amazon ECS

[ductrieu]

I think you forgot config http_listen_port. after start grafana-agent you should run this command: grafana-agent -config.file = agent.yaml -http_listen_port = 12345

[openPablo]

I ran into the same issue trying to deploy grafana agent on ECS.

Version v0.23.0 seems to be the last version where the ready and healthy checkpoints aren't broken.

docker run -p 12345:12345 --rm grafana/agent:v0.23.0
curl http://localhost:12345/-/healthy

Agent is Healthy.

docker run -p 12345:12345 --rm grafana/agent:v0.24.0
curl http://localhost:12345/-/healthy

curl: (56) Recv failure: Connection reset by peer

Documentation says:
(Note that scraping service mode is a requirement for the config management API, however this is not a prerequisite for the Agent API or Ready/Healthy API)
So I would expect /-/healthy and to work like v0.23.0.

@ductrieu When v0.32.0 starts it says msg="server listening on addresses" http=127.0.0.1:12345 with default configuration.

[tpaschalis]

Hey folks, Grafana Agent dev here. I'm trying to understand if it's an ECS-specific problem, since the endpoint seems to be working properly in all versions I've tested (including v0.24.0).

$ docker run -p 12345:12345 --rm grafana/agent:v0.32.1
Unable to find image 'grafana/agent:v0.32.1' locally
v0.32.1: Pulling from grafana/agent
...

$ curl localhost:12345/-/healthy
Agent is Healthy.

Are you by any chance running mounting a large WAL from previous Agent runs? The agent might need some time to replay the WAL and then start scraping for metrics and receiving requests.

[openPablo]

Thanks for your fast reply!
I'm not mounting any volumes with localhost tests or in ECS.
However in my local tests I'm not adding a custom config yaml, so it tries to push to (edit) the not existing http://localhost:9009/api/prom/push.
Is this the cause?
logs:

pablo@sol ~ ()$ docker run -p 12345:12345 --rm grafana/agent:v0.32.1
ts=2023-04-06T08:11:30.790715686Z caller=server.go:191 level=info msg="server listening on addresses" http=127.0.0.1:12345 grpc=127.0.0.1:12346 http_tls_enabled=false grpc_tls_enabled=false
ts=2023-04-06T08:11:30.790960441Z caller=node.go:85 level=info agent=prometheus component=cluster msg="applying config"
ts=2023-04-06T08:11:30.791055672Z caller=remote.go:180 level=info agent=prometheus component=cluster msg="not watching the KV, none set"
ts=2023-04-06T08:11:30.791549821Z caller=zapadapter.go:78 level=info component=traces msg="Traces Logger Initialized"
ts=2023-04-06T08:11:30.792863829Z caller=reporter.go:103 level=info msg="running usage stats reporter"
ts=2023-04-06T08:11:30.794499027Z caller=wal.go:199 level=info agent=prometheus instance=72dc368efd128e2b2086a1f430082fee msg="replaying WAL, this may take a while" dir=/etc/agent/data/72dc368efd128e2b2086a1f430082fee/wal
ts=2023-04-06T08:11:30.794569631Z caller=wal.go:246 level=info agent=prometheus instance=72dc368efd128e2b2086a1f430082fee msg="WAL segment loaded" segment=0 maxSegment=0
ts=2023-04-06T08:11:30.795370453Z caller=dedupe.go:112 agent=prometheus instance=72dc368efd128e2b2086a1f430082fee component=remote level=info remote_name=72dc36-990440 url=http://localhost:9009/api/prom/push msg="Starting WAL watcher" queue=72dc36-990440
ts=2023-04-06T08:11:30.795381545Z caller=dedupe.go:112 agent=prometheus instance=72dc368efd128e2b2086a1f430082fee component=remote level=info remote_name=72dc36-990440 url=http://localhost:9009/api/prom/push msg="Starting scraped metadata watcher"
ts=2023-04-06T08:11:30.795495952Z caller=dedupe.go:112 agent=prometheus instance=72dc368efd128e2b2086a1f430082fee component=remote level=info remote_name=72dc36-990440 url=http://localhost:9009/api/prom/push msg="Replaying WAL" queue=72dc36-990440
ts=2023-04-06T08:11:36.836354465Z caller=dedupe.go:112 agent=prometheus instance=72dc368efd128e2b2086a1f430082fee component=remote level=info remote_name=72dc36-990440 url=http://localhost:9009/api/prom/push msg="Done replaying WAL" duration=6.0409153s
ts=2023-04-06T08:11:40.827511401Z caller=dedupe.go:112 agent=prometheus instance=72dc368efd128e2b2086a1f430082fee component=remote level=warn remote_name=72dc36-990440 url=http://localhost:9009/api/prom/push msg="Failed to send batch, retrying" err="Post \"http://localhost:9009/api/prom/push\": dial tcp 127.0.0.1:9009: connect: connection refused"
ts=2023-04-06T08:12:43.452340884Z caller=dedupe.go:112 agent=prometheus instance=72dc368efd128e2b2086a1f430082fee component=remote level=warn remote_name=72dc36-990440 url=http://localhost:9009/api/prom/push msg="Failed to send batch, retrying" err="Post \"http://localhost:9009/api/prom/push\": dial tcp 127.0.0.1:9009: connect: connection refused"

[rfratto]

Sorry for the confusion here. v0.23.0 was the last version where the default listen address was 0.0.0.0:12345, which was changed afterwards to 127.0.0.1:12345 for security reasons.

To expose the API to outside of the container, you need to pass -server.http.address=0.0.0.0:12345 as an argument. I'm not sure why it worked for Paschalis (it may be his Docker network configuration), but passing that flag should get everything working for you again.

[JCastrejonE]

Thanks! This argument makes the API works as expected

[llamafilm]

I like the idea of restricting this to localhost only, but it seems like curl is not installed in the container so how is this intended to be used?