vault kubernetes auth doesn't work #35

gricuk · 2022-02-15T18:14:27Z

Hi, I'm trying to use your vault integration with k8s auth, but it doesn't work.
I got the following error

2022-02-15 18:05:38,790 INFO Initializing client with mode [KUBERNETES] (io.lenses.connect.secrets.providers.VaultSecretProvider) [main]
2022-02-15 18:05:39,312 ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed) [main]
com.bettercloud.vault.VaultException: Vault responded with HTTP status code: 400
Response body: {"errors":["missing client token"]}

	at com.bettercloud.vault.api.Auth.loginByJwt(Auth.java:1045)
	at com.bettercloud.vault.api.Auth.loginByKubernetes(Auth.java:1113)
	at io.lenses.connect.secrets.providers.VaultHelper.$anonfun$createClient$5(VaultHelper.scala:84)
	at scala.Option.map(Option.scala:230)
	at io.lenses.connect.secrets.providers.VaultHelper.createClient(VaultHelper.scala:81)
	at io.lenses.connect.secrets.providers.VaultHelper.createClient$(VaultHelper.scala:19)
	at io.lenses.connect.secrets.providers.VaultSecretProvider.createClient(VaultSecretProvider.scala:31)
	at io.lenses.connect.secrets.providers.VaultSecretProvider.configure(VaultSecretProvider.scala:43)
	at org.apache.kafka.common.config.AbstractConfig.instantiateConfigProviders(AbstractConfig.java:572)
	at org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:515)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:107)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:129)
	at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:452)
	at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:405)
	at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:95)
	at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)

I've created a service account and role binding, also I've created k8s auth in vault.
Also I've tested that auth works inside container:

curl -k --request POST --data '{"jwt": "jwt-from-service-account", "role": "kv-k8s-test-kafka-connect-cluster-role"}' https://vault_addr:8200/v1/auth/k8s-cluster/login
{
  "request_id": "5433b4ac-ab83-8c84-2f52-e95cff9af9eb",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": null,
  "wrap_info": null,
  "warnings": null,
  "auth": {
    "client_token": "s.token",
    "accessor": "eKnQVOKH7H",
    "policies": [
      "default",
      "policy-kv-k8s-kafka-connect-cluster-read"
    ],

I don't understand why plugin can't get a secret with missing client token error

The text was updated successfully, but these errors were encountered:

gricuk · 2022-02-16T11:51:30Z

I've found a problem. The plugin works, but it uses the default kubernetes auth for login - auth/kubernetes/login . In case if you have a non-default Kubernetes auth name - it won't work

ifurdi-interhyp · 2022-03-02T11:24:16Z

Hi, I stumbled upon the same issue (our path is a different one than default auth/kubernetes/login). @gricuk Did you manage to solve this issue? I have additional issue runing this plugin with version 7.0.1, which version are you using?

gricuk · 2022-04-12T10:53:54Z

Hi, I stumbled upon the same issue (our path is a different one than default auth/kubernetes/login). @gricuk Did you manage to solve this issue? I have additional issue runing this plugin with version 7.0.1, which version are you using?

Hi, we have forked repo and added logic for setting non-default auth .

davidsloan · 2023-03-19T14:41:24Z

This looks to be the same issue as #46. The reporter is pursuing an approach that will resolve this issue. Stay tuned.

mcl085 · 2023-06-05T13:27:26Z

Hi, I'm trying to implement the plugin with Vault but it returns the error:

com.bettercloud.vault.VaultException: Vault responded with HTTP status code: 403
Response body: {"errors":["permission denied"]}

My k8s uses a custom path as well. I tried to use the variable "KUBERNETES_AUTH_PATH" using the path as value, but the kafka connect ignores the values on start up.

Is there anything that I have missed? I'm struggling a lot to make it works.

mcl085 · 2023-06-05T13:32:01Z

My config-map using the variable with the custom path declared:

CONNECT_CONFIG_PROVIDERS_VAULT_PARAM_KUBERNETES_AUTH_PATH: "auth/my_path"

The values logged on startup. Note that ignores the variable value above

[2023-06-05 10:19:47,840] INFO VaultProviderConfig values: 
	app.role.id = f1ef9a30-2438-96a3-177d-1e6faaf9fb2a
	app.role.secret.id = null
	aws.mount = aws
	aws.request.body = null
	aws.request.headers = null
	aws.request.url = null
	aws.role = null
	cert.mount = cert
	file.dir = /connector-files/vault
	file.write = false
	gcp.jwt = null
	gcp.role = null
	github.mount = github
	github.token = null
	jwt = null
	jwt.provider = null
	jwt.role = null
	kubernetes.role = my_k8s_role
	kubernetes.token.path = /var/run/secrets/kubernetes.io/serviceaccount/token
	ldap.mount = ldap
	ldap.password = null
	ldap.username = null
	mount = userpass
	password = null
	secret.default.ttl = 0
	token.renewal.ms = 600000
	username = null
	vault.addr = https://my_vault.intranet
	vault.auth.method = kubernetes
	vault.client.pem = 
	vault.engine.version = 2
	vault.namespace = 
	vault.pem = 
	vault.ssl.keystore.location = 
	vault.ssl.keystore.password = [hidden]
	vault.ssl.truststore.location = /etc/pki/ca-trust/extracted/java/cacerts
	vault.token = null
 (io.lenses.connect.secrets.config.VaultProviderConfig:376)
[2023-06-05 10:19:48,243] INFO Configuring keystore at [/etc/pki/ca-trust/extracted/java/cacerts] (io.lenses.connect.secrets.providers.VaultSecretProvider:152)
[2023-06-05 10:19:48,251] INFO Setting engine version to 2 (io.lenses.connect.secrets.providers.VaultSecretProvider:34)
[2023-06-05 10:19:48,253] INFO Initializing client with mode [KUBERNETES] (io.lenses.connect.secrets.providers.VaultSecretProvider:40)
[2023-06-05 10:19:49,163] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed:86)
com.bettercloud.vault.VaultException: Vault responded with HTTP status code: 403
Response body: {"errors":["permission denied"]}

	at com.bettercloud.vault.api.Auth.loginByJwt(Auth.java:1045)
	at com.bettercloud.vault.api.Auth.loginByKubernetes(Auth.java:1113)
	at io.lenses.connect.secrets.providers.VaultHelper.$anonfun$createClient$5(VaultHelper.scala:86)
	at scala.Option.map(Option.scala:242)
	at io.lenses.connect.secrets.providers.VaultHelper.createClient(VaultHelper.scala:83)
	at io.lenses.connect.secrets.providers.VaultHelper.createClient$(VaultHelper.scala:22)
	at io.lenses.connect.secrets.providers.VaultSecretProvider.createClient(VaultSecretProvider.scala:33)
	at io.lenses.connect.secrets.providers.VaultSecretProvider.configure(VaultSecretProvider.scala:47)
	at org.apache.kafka.common.config.AbstractConfig.instantiateConfigProviders(AbstractConfig.java:576)
	at org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:519)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:112)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:133)
	at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:366)
	at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:405)
	at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:95)
	at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)

davidsloan · 2023-06-05T13:32:27Z

@mcl085 have you taken and built the latest code from master branch? We are yet to release the latest enhancements for vault

mcl085 · 2023-06-05T13:47:51Z

Hi @davidsloan

I only updated from release 2.1.6 to 2.2.0 and built a new docker image with this new tag

davidsloan · 2023-06-05T13:52:43Z

Hi @davidsloan

I only updated from release 2.1.6 to 2.2.0 and built a new docker image with this new tag

#51

@mcl085 I think you need this change. A release has not been made yet hence my comment about compiling from source. Shout if you need further help testing it.

philipsabri · 2023-06-09T13:19:14Z

@mcl085 I think you need this change. A release has not been made yet hence my comment about compiling from source. Shout if you need further help testing it.

Any idea when an official release is coming?

andrewstevenson · 2023-10-25T10:28:47Z

@philipsabri We will be releasing within the next few days. Sorry for the delay.

davidsloan · 2023-11-01T15:43:59Z

This is released in 2.3.0

davidsloan added the pending-release label Nov 1, 2023

davidsloan closed this as completed Nov 1, 2023

davidsloan removed the pending-release label Nov 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vault kubernetes auth doesn't work #35

vault kubernetes auth doesn't work #35

gricuk commented Feb 15, 2022

gricuk commented Feb 16, 2022

ifurdi-interhyp commented Mar 2, 2022

gricuk commented Apr 12, 2022 •

edited

Loading

davidsloan commented Mar 19, 2023

mcl085 commented Jun 5, 2023

mcl085 commented Jun 5, 2023 •

edited

Loading

davidsloan commented Jun 5, 2023

mcl085 commented Jun 5, 2023

davidsloan commented Jun 5, 2023 •

edited

Loading

philipsabri commented Jun 9, 2023

andrewstevenson commented Oct 25, 2023

davidsloan commented Nov 1, 2023

vault kubernetes auth doesn't work #35

vault kubernetes auth doesn't work #35

Comments

gricuk commented Feb 15, 2022

gricuk commented Feb 16, 2022

ifurdi-interhyp commented Mar 2, 2022

gricuk commented Apr 12, 2022 • edited Loading

davidsloan commented Mar 19, 2023

mcl085 commented Jun 5, 2023

mcl085 commented Jun 5, 2023 • edited Loading

davidsloan commented Jun 5, 2023

mcl085 commented Jun 5, 2023

davidsloan commented Jun 5, 2023 • edited Loading

philipsabri commented Jun 9, 2023

andrewstevenson commented Oct 25, 2023

davidsloan commented Nov 1, 2023

gricuk commented Apr 12, 2022 •

edited

Loading

mcl085 commented Jun 5, 2023 •

edited

Loading

davidsloan commented Jun 5, 2023 •

edited

Loading