Hardeneks doesn't work through the SSH tunnel

[elibogomolnyi]

We are trying to use hardeneks for hardening our cluster. We connect to the cluster by the ssh tunnel through the bastion machine. Here is how we connect to our cluster.

Connecting to the AWS account admin user by sso

export AWS_PROFILE=MainAdmin
export AWS_REGION=eu-west-1
export K8S_AUTH_PROXY=""
export NO_PROXY=""
export HTTP_PROXY=""
export HTTPS_PROXY=""
aws configure sso

Creating an SSH tunnel and connection to the cluster

ssh-add ~/keys/key/our-key
ssh -L 8888:localhost:8888 -q -o StrictHostKeyChecking=no -C -N [email protected] &
export K8S_AUTH_PROXY=http://localhost:8888
export NO_PROXY=*.okta.com
export HTTP_PROXY=http://localhost:8888
export HTTPS_PROXY=http://localhost:8888
aws eks --region eu-west-1 update-kubeconfig --name our_cluster

After it, we can run all the kubectl-related commands and fully manage resources in our cluster.

But when we go with hardeneks, it first gets stuck here.

*  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  * HARDENEKS *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *  *
You are operating at eu-west-1
You context is arn:aws:eks:eu-west-1:717343414241:cluster/our_cluster
Your cluster name is our_cluster
You are using /private/tmp/.venv/lib/python3.9/site-packages/hardeneks/config.yaml as your config file

And then it fails with this error (the host was changed intensionally to hide the real DNS):

MaxRetryError: HTTPSConnectionPool(host='a9276e4d543d078f345a64b343d23eb1.gr7.eu-west-1.eks.amazonaws.com', port=443): Max retries exceeded with url:
/api/v1/namespaces (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x1092de490>: Failed to establish a new connection: [Errno 60]
Operation timed out'))

How can I configure the hardeneks to make requests through the SSH tunnel? I think that it is a typical issue since most clusters are not publicly exposed.

[dorukozturk]

Hi @elibogomolnyi, thank you very much for trying hardeneks and creating an issue. I will try to reproduce your issue on my end.

Hardeneks uses kubernetes python api underneath (https://github.com/kubernetes-client/python).

I saw similar issues before from that library where people are able to kubectl but sometimes when they try the same operations through the python api it does not work:
kubernetes-client/python#1767

Can you try to add the --insecure-skip-tls-verify flag to see if this about TLS verification?

[elibogomolnyi]

Hi @dorukozturk, thank you for the fast response. I also tried the --insecure-skip-tls-verify flag, which didn't help.

[dorukozturk]

Thank you. Ok I will try to reproduce your issue on my end will fix it soon.

[dorukozturk]

I did some research:

Seems like Kubernetes Python API does not respect the proxy environment vars such as:
export NO_PROXY=""
export HTTP_PROXY=""
export HTTPS_PROXY=""

But in the configuration there is a proxy attribute we can set.

I will try to create a proxy for an EKS cluster and see if that fix helps.

I will keep you posted

[dorukozturk]

Hi @elibogomolnyi just to give you an update:

Kubernetes python api does not work with neither the HTTP_PROXY environment variable nor with the proxy section in the kubeconfig file unfortunately. There are couple of PRs open.

kubernetes-client/python#1862
kubernetes-client/python#1863
kubernetes-client/python#1967

Ideally I would like to avoid doing a hack on the hardeneks side but I might have to. I asked for an update on those issues. Let's see what the maintainers say.

Is there a way that you can run hardeneks from a jumpbox or a bastion host while these issues are worked on?