Bad Gateway - Not able to add Cortex data source #3702
Replies: 4 comments 5 replies
-
|
Hey @vicmania2007, One thing I noticed is you are using port 8080 for HTTP requests. However, you do not define the port in the nginx config. You can see in the official Cortex helm chart here that the ports should be explicitly defined. Could you update your nginx config to following and see if that works: kind: ConfigMap
apiVersion: v1
metadata:
name: cortex-nginx
namespace: default
labels:
app: cortex-nginx
chart: cortex
release: cortex
heritage: Tiller
data:
nginx.conf: |-
worker_processes 5; ## Default: 1
error_log /dev/stderr;
pid /tmp/nginx.pid;
worker_rlimit_nofile 8192;
events {
worker_connections 4096; ## Default: 1024
}
http {
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] $status '
'"$request" $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /dev/stderr main;
sendfile on;
tcp_nopush on;
resolver coredns.kube-system.svc.cluster.local;
server { # simple reverse-proxy
listen 80;
proxy_set_header X-Scope-OrgID 0;
location = /healthz {
return 200 'alive';
}
location = /api/prom/push {
proxy_pass http://cortex-distributor.default.svc.cluster.local:8080$request_uri;
}
location = /ring {
proxy_pass http://cortex-distributor.default.svc.cluster.local:8080$request_uri;
}
location = /config {
proxy_pass http://cortex-distributor.default.svc.cluster.local:8080$request_uri;
}
location = /all_user_stats {
proxy_pass http://cortex-distributor.default.svc.cluster.local:8080$request_uri;
}
location ~ /api/prom/.* {
proxy_pass http://cortex-querier.default.svc.cluster.local:8080$request_uri;
}
}
} |
Beta Was this translation helpful? Give feedback.
-
|
@vicmania2007 Where are you running Grafana? You said you port-forwarded Nginx to your local and queried it from Grafana. Does your local grafana have access to the ports running on your node? For instance, if you are running Grafana in a container it may not have access unless you specify it. Second, have you tried port-forwarding the queriers to your local machine and querying them instead of Nginx? |
Beta Was this translation helpful? Give feedback.
-
When i hit localhost:31500 i see this screen
and when i hit localhost:31500/config
Do i have to query from a different port or is that a different URL for querying? |
Beta Was this translation helpful? Give feedback.
-
|
I tried the following, but still facing the same issues.
When i was doing above i noticed that querier was giving out weird logs. I couldnt understand much but thought it might be connected to store-gateway. Checking the store-gateway logs, i found this. I created a new Azure container and redeployed the store-gateway, after which I didnt see this error message.
|
Beta Was this translation helpful? Give feedback.
-
|
Logs of querier, stor-gateway and ingester. |
Beta Was this translation helpful? Give feedback.
-
|
I was facing similar issue earlier while doing a production setup. But it was to do with the wrong URL in Grafana. Can you check if you are after send the same request to querier and get the result using curl. If yes then the issue is with gateway. Enable debug mode and see if the request you are sending from Grafana reaches gateway. You should be able to see that in Gateway logs. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Chinmay, i removed nginx layer for troubleshooting and hit the querier service with curl instead of grafana. The traffic is going to the service and also to the querier pod, but that particular URL (/api/prom)is giving 404 page not found. Same behavior even with nginx in between. I havent started using gateway as of yet, if you are referring to https://github.com/rewe-digital/cortex-gateway |
Beta Was this translation helpful? Give feedback.
-
|
Start your querier in debug mode, And share the the logs when you hit the /api/prom url. |
Beta Was this translation helpful? Give feedback.
-
|
What you want is localhost:8080/prometheus The default https://cortexmetrics.io/docs/getting-started/#single-instance-single-process |
Beta Was this translation helpful? Give feedback.
-
I have my block storage cortex setup with the write path working and data getting into Azure blob. Even though the querier is working, i am not able to add cortex as data source into grafana to validate the read path. I dont know what I am missing.
To Test:
I have made the nginx service as nodeport and performing port forwarding, so i can test locally.
akc port-forward service/cortex 8080:8080 Forwarding from 127.0.0.1:8080 -> 80 Forwarding from [::1]:8080 -> 80 Handling connection for 8080 Handling connection for 8080 Handling connection for 8080 Handling connection for 8080 Handling connection for 8080 Handling connection for 8080Grafana Error:
1.This is my config:
2. Querier logs:
level=info ts=2021-01-15T23:22:33.275253807Z caller=main.go:187 msg="Starting Cortex" version="(version=1.5.0, branch=HEAD, revision=86d95e7)" level=info ts=2021-01-15T23:22:33.275478666Z caller=server.go:225 http=[::]:8080 grpc=[::]:9095 msg="server listening on addresses" level=info ts=2021-01-15T23:22:33.431412647Z caller=worker.go:65 msg="no address specified, not starting worker" level=info ts=2021-01-15T23:22:33.431577807Z caller=module_service.go:58 msg=initialising module=server level=info ts=2021-01-15T23:22:33.431611178Z caller=module_service.go:58 msg=initialising module=memberlist-kv level=info ts=2021-01-15T23:22:33.43167128Z caller=module_service.go:58 msg=initialising module=ring level=info ts=2021-01-15T23:22:33.431721627Z caller=module_service.go:58 msg=initialising module=store-queryable level=info ts=2021-01-15T23:22:33.431852617Z caller=module_service.go:58 msg=initialising module=distributor-service level=info ts=2021-01-15T23:22:35.628224319Z caller=fetcher.go:453 org_id=fake component=block.BaseFetcher msg="successfully synchronized block metadata" duration=2.170020084s cached=153 returned=153 partial=0 level=info ts=2021-01-15T23:22:36.981312096Z caller=cortex.go:355 msg="Cortex started" level=info ts=2021-01-15T23:27:41.617091687Z caller=fetcher.go:453 org_id=fake component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.855914346s cached=163 returned=163 partial=0 level=info ts=2021-01-15T23:32:44.310315136Z caller=fetcher.go:453 org_id=fake component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.828943292s cached=173 returned=173 partial=03. Nginx configmap:
Beta Was this translation helpful? Give feedback.
All reactions