Skip to content

Latest commit

 

History

History
288 lines (225 loc) · 11.3 KB

README.md

File metadata and controls

288 lines (225 loc) · 11.3 KB
Raw

Create EKS cluster

Amazon EKS

Before starting with the main content, it's necessary to provision the Amazon EKS in AWS.

Use the MY_DOMAIN variable containing domain and LETSENCRYPT_ENVIRONMENT variable. The LETSENCRYPT_ENVIRONMENT variable should be one of:

  • staging - Let’s Encrypt will create testing certificate (not valid)

  • production - Let’s Encrypt will create valid certificate (use with care)

export MY_DOMAIN=${MY_DOMAIN:-mylabs.dev}
export LETSENCRYPT_ENVIRONMENT=${LETSENCRYPT_ENVIRONMENT:-staging}
echo "${MY_DOMAIN} | ${LETSENCRYPT_ENVIRONMENT}"

Prepare the local working environment

::: tip You can skip these steps if you have all the required software already installed. :::

Install necessary software:

test -x /usr/bin/apt && \
apt update -qq && \
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq awscli curl docker.io freerdp-x11 gettext-base git gnupg2 jq ldap-utils openssh-client python3-pip sudo wget > /dev/null && \
pip3 install --quiet ansible boto3 pywinrm

Install kubectl binary:

if [ ! -x /usr/local/bin/kubectl ]; then
  sudo curl -s -Lo /usr/local/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
  sudo chmod a+x /usr/local/bin/kubectl
fi

Install eksctl:

if [ ! -x /usr/local/bin/eksctl ]; then
  curl -s -L "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_Linux_amd64.tar.gz" | sudo tar xz -C /usr/local/bin/
fi

Install AWS IAM Authenticator for Kubernetes:

if [ ! -x /usr/local/bin/aws-iam-authenticator ]; then
  sudo curl -s -Lo /usr/local/bin/aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.12.7/2019-03-27/bin/linux/amd64/aws-iam-authenticator
  sudo chmod a+x /usr/local/bin/aws-iam-authenticator
fi

Configure AWS

Authorize to AWS using AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html

aws configure
...

Create DNS zone:

aws route53 create-hosted-zone --name ${MY_DOMAIN} --caller-reference ${MY_DOMAIN}

Use your domain registrar to change the nameservers for your zone (for example "mylabs.dev") to use the Amazon Route 53 nameservers. Here is the way how you can find out the the Route 53 nameservers:

aws route53 get-hosted-zone --id $(aws route53 list-hosted-zones --query "HostedZones[?Name==\`${MY_DOMAIN}.\`].Id" --output text) --query "DelegationSet.NameServers"

Create policy allowing the cert-manager to change Route 53 settings. This will allow cert-manager to generate wildcard SSL certificates by Let's Encrypt certificate authority.

aws iam create-policy \
  --policy-name ${USER}-AmazonRoute53Domains-cert-manager \
  --description "Policy required by cert-manager to be able to modify Route 53 when generating wildcard certificates using Lets Encrypt" \
  --policy-document file://files/route_53_change_policy.json \
| jq

Output:

{
    "Policy": {
        "PolicyName": "pruzicka-AmazonRoute53Domains-cert-manager",
        "PolicyId": "ANPA36ZNO4Q4MTW5T5ZLX",
        "Arn": "arn:aws:iam::822044714040:policy/pruzicka-AmazonRoute53Domains-cert-manager",
        "Path": "/",
        "DefaultVersionId": "v1",
        "AttachmentCount": 0,
        "IsAttachable": true,
        "CreateDate": "2019-06-05T11:16:58Z",
        "UpdateDate": "2019-06-05T11:16:58Z"
    }
}

Create user which will use the policy above allowing the cert-manager to change Route 53 settings:

aws iam create-user --user-name ${USER}-eks-cert-manager-route53 | jq && \
POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName==\`${USER}-AmazonRoute53Domains-cert-manager\`].{ARN:Arn}" --output text) && \
aws iam attach-user-policy --user-name "${USER}-eks-cert-manager-route53" --policy-arn $POLICY_ARN && \
aws iam create-access-key --user-name ${USER}-eks-cert-manager-route53 > $HOME/.aws/${USER}-eks-cert-manager-route53-${MY_DOMAIN} && \
export EKS_CERT_MANAGER_ROUTE53_AWS_ACCESS_KEY_ID=$(awk -F\" "/AccessKeyId/ { print \$4 }" $HOME/.aws/${USER}-eks-cert-manager-route53-${MY_DOMAIN}) && \
export EKS_CERT_MANAGER_ROUTE53_AWS_SECRET_ACCESS_KEY=$(awk -F\" "/SecretAccessKey/ { print \$4 }" $HOME/.aws/${USER}-eks-cert-manager-route53-${MY_DOMAIN})

Output:

{
    "User": {
        "Path": "/",
        "UserName": "pruzicka-eks-cert-manager-route53",
        "UserId": "AIDA36ZNO4Q4NSQJW6T7V",
        "Arn": "arn:aws:iam::822044714040:user/pruzicka-eks-cert-manager-route53",
        "CreateDate": "2019-06-05T11:16:59Z"
    }
}

The AccessKeyId and SecretAccessKey is need for creating the ClusterIssuer definition for cert-manager.

Create Amazon EKS

EKS

Generate SSH keys if not exists:

test -f $HOME/.ssh/id_rsa || ( install -m 0700 -d $HOME/.ssh && ssh-keygen -b 2048 -t rsa -f $HOME/.ssh/id_rsa -q -N "" )

Clone the k8s-harbor Git repository if it wasn't done already:

[ ! -d .git ] && git clone --quiet https://github.com/ruzickap/k8s-harbor && cd k8s-harbor

Create Amazon EKS in AWS by using eksctl. It's a tool from Weaveworks based on official AWS CloudFormation templates which will be used to launch and configure our EKS cluster and nodes.

eksctl

eksctl create cluster \
--name=${USER}-k8s-harbor \
--tags "Application=Harbor,Owner=${USER},Environment=Test,Division=Services" \
--region=eu-central-1 \
--node-type=t3.large \
--ssh-access \
--ssh-public-key $HOME/.ssh/id_rsa.pub \
--node-ami=auto \
--node-labels "Application=Harbor,Owner=${USER},Environment=Test,Division=Services" \
--kubeconfig=kubeconfig.conf

Output:

[ℹ]  using region eu-central-1
[ℹ]  setting availability zones to [eu-central-1c eu-central-1b eu-central-1a]
[ℹ]  subnets for eu-central-1c - public:192.168.0.0/19 private:192.168.96.0/19
[ℹ]  subnets for eu-central-1b - public:192.168.32.0/19 private:192.168.128.0/19
[ℹ]  subnets for eu-central-1a - public:192.168.64.0/19 private:192.168.160.0/19
[ℹ]  nodegroup "ng-d1b535b2" will use "ami-0b7127e7a2a38802a" [AmazonLinux2/1.13]
[ℹ]  using SSH public key "/home/pruzicka/.ssh/id_rsa.pub" as "eksctl-pruzicka-k8s-harbor-nodegroup-ng-d1b535b2-a3:84:e4:0d:af:5f:c8:40:da:71:68:8a:74:c7:ba:16"
[ℹ]  using Kubernetes version 1.13
[ℹ]  creating EKS cluster "pruzicka-k8s-harbor" in "eu-central-1" region
[ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
[ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=eu-central-1 --name=pruzicka-k8s-harbor'
[ℹ]  2 sequential tasks: { create cluster control plane "pruzicka-k8s-harbor", create nodegroup "ng-d1b535b2" }
[ℹ]  building cluster stack "eksctl-pruzicka-k8s-harbor-cluster"
[ℹ]  deploying stack "eksctl-pruzicka-k8s-harbor-cluster"
[ℹ]  building nodegroup stack "eksctl-pruzicka-k8s-harbor-nodegroup-ng-d1b535b2"
[ℹ]  --nodes-min=2 was set automatically for nodegroup ng-d1b535b2
[ℹ]  --nodes-max=2 was set automatically for nodegroup ng-d1b535b2
[ℹ]  deploying stack "eksctl-pruzicka-k8s-harbor-nodegroup-ng-d1b535b2"
[✔]  all EKS cluster resource for "pruzicka-k8s-harbor" had been created
[✔]  saved kubeconfig as "kubeconfig.conf"
[ℹ]  adding role "arn:aws:iam::822044714040:role/eksctl-pruzicka-k8s-harbor-nodegr-NodeInstanceRole-A4XWMWDV73D9" to auth ConfigMap
[ℹ]  nodegroup "ng-d1b535b2" has 0 node(s)
[ℹ]  waiting for at least 2 node(s) to become ready in "ng-d1b535b2"
[ℹ]  nodegroup "ng-d1b535b2" has 2 node(s)
[ℹ]  node "ip-192-168-56-161.eu-central-1.compute.internal" is ready
[ℹ]  node "ip-192-168-9-2.eu-central-1.compute.internal" is ready
[ℹ]  kubectl command should work with "kubeconfig.conf", try 'kubectl --kubeconfig=kubeconfig.conf get nodes'
[✔]  EKS cluster "pruzicka-k8s-harbor" in "eu-central-1" region is ready

EKS Architecture

Create CloudFormation stack with Windows Server 2016, which will serve as Active Directory to use LDAP connection from Harbor:

ansible-playbook --connection=local -i "127.0.0.1," files/ansible/aws_windows_server_2016.yml

Output:

...
PLAY RECAP *********************************************************************
127.0.0.1                  : ok=6    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
winad01.mylabs.dev         : ok=22   changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

You should be able to access Windows Server using RDP:

xfreerdp /u:Administrator /p:really_long_secret_windows_password /size:1440x810 -wallpaper /cert-ignore /dynamic-resolution /v:winad01.${MY_DOMAIN} &> /dev/null &

Windows desktop should appear:

Windows Server

If you check the AD Users you should see users aduser{01..06} distributed into three groups adgroup{01.03} with password admin.

Check if the new EKS cluster is available:

export KUBECONFIG=$PWD/kubeconfig.conf
kubectl get nodes -o wide

Output:

NAME                                              STATUS   ROLES    AGE   VERSION              INTERNAL-IP      EXTERNAL-IP     OS-IMAGE         KERNEL-VERSION                  CONTAINER-RUNTIME
ip-192-168-56-161.eu-central-1.compute.internal   Ready    <none>   46m   v1.13.7-eks-c57ff8   192.168.56.161   54.93.96.15     Amazon Linux 2   4.14.128-112.105.amzn2.x86_64   docker://18.6.1
ip-192-168-9-2.eu-central-1.compute.internal      Ready    <none>   46m   v1.13.7-eks-c57ff8   192.168.9.2      18.196.16.153   Amazon Linux 2   4.14.128-112.105.amzn2.x86_64   docker://18.6.1

EKS High Level

Both worker nodes should be accessible via SSH:

for EXTERNAL_IP in $(kubectl get nodes --output=jsonpath="{.items[*].status.addresses[?(@.type==\"ExternalIP\")].address}"); do
  echo "*** ${EXTERNAL_IP}"
  ssh -q -n -o StrictHostKeyChecking=no -l ec2-user ${EXTERNAL_IP} uptime
done

Output:

*** 54.93.96.15
 10:16:43 up 48 min,  0 users,  load average: 1.03, 0.47, 0.25
*** 18.196.16.153
 10:16:43 up 48 min,  0 users,  load average: 0.64, 0.91, 0.61

At the end of the output you should see 2 IP addresses which should be accessible by SSH using your public key ~/.ssh/id_rsa.pub.