All posts in Web Development

The Kubernetes's Cluster Autoscaler automatically adjusts the number of nodes in your cluster when pods fail or are rescheduled onto other nodes. Here we're going to deploy the AWS implementation that implements the decisions of Cluster Autoscaler by communicating with AWS products and services such as Amazon EC2.

ref:
https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html

Configure IAM Permissions

If your existing node groups were created with eksctl create nodegroup --asg-access, then this policy already exists and you can skip this step.

in cluster-autoscaler-policy-staging.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:ResourceTag/k8s.io/cluster-autoscaler/perp-staging": "owned"
                }
            }
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "ec2:DescribeLaunchTemplateVersions"
            ],
            "Resource": "*"
        }
    ]
}

aws --profile perp iam create-policy \
  --policy-name AmazonEKSClusterAutoscalerPolicyStaging \
  --policy-document file://cluster-autoscaler-policy-staging.json

eksctl --profile=perp create iamserviceaccount \
  --cluster=perp-staging \
  --namespace=kube-system \
  --name=cluster-autoscaler \
  --attach-policy-arn=arn:aws:iam::xxx:policy/AmazonEKSClusterAutoscalerPolicy \
  --override-existing-serviceaccounts \
  --approve

ref:
https://docs.aws.amazon.com/eks/latest/userguide/autoscaling.html

Deploy Cluster Autoscaler

Download the deployment yaml of cluster-autoscaler:

curl -o cluster-autoscaler-autodiscover.yaml \
https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml

Before you apply the file, it's recommended to check whether the version of cluster-autoscaler matches the Kubernetes major and minor version of your cluster. Find the version number on GitHub releases.

kubectl apply -f cluster-autoscaler-autodiscover.yaml
# or
kubectl set image deployment cluster-autoscaler \
  -n kube-system \
  cluster-autoscaler=k8s.gcr.io/autoscaling/cluster-autoscaler:v1.21.3

Do some tweaks:

• --balance-similar-node-groups ensures that there is enough available compute across all availability zones.
• --skip-nodes-with-system-pods=false ensures that there are no problems with scaling to zero.

kubectl patch deployment cluster-autoscaler \
  -n kube-system \
  -p '{"spec":{"template":{"metadata":{"annotations":{"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"}}}}}'

kubectl -n kube-system edit deployment.apps/cluster-autoscaler
# change the command to the following:
# spec:
#   containers:
#   - command
#     - ./cluster-autoscaler
#     - --v=4
#     - --stderrthreshold=info
#     - --cloud-provider=aws
#     - --skip-nodes-with-local-storage=false
#     - --expander=least-waste
#     - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/perp-staging
#     - --balance-similar-node-groups
#     - --skip-nodes-with-system-pods=false

ref:
https://aws.github.io/aws-eks-best-practices/cluster-autoscaling/
https://github.com/kubernetes/autoscaler/tree/master/charts/cluster-autoscaler#additional-configuration

kubernetes-external-secrets allows you to use external secret management systems, like AWS Secrets Manager, to securely add Secrets in Kubernetes, so Pods can access Secrets normally.

ref:
https://github.com/external-secrets/kubernetes-external-secrets

AWS Secret Manager

For instance, we create a secret named YOUR_SECRET on AWS Secret Manager in the same region as our EKS cluster, using DefaultEncryptionKey as the encryption key. The content of the secret entity look like:

{
  "KEY_1": "VALUE_1",
  "KEY_2": "VALUE_2",
}

We can retrieve the secret value:

aws secretsmanager get-secret-value --profile=perp \
--region ap-northeast-1 \
--secret-id YOUR_SECRET

kubernetes-external-secrets

For kubernetes-external-secrets to work properly, it must be granted access to AWS Secrets Manager. To achieve that, we need to create an IAM role for kubernetes-external-secrets' service account.

ref:
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

Configure Secrets Backends

Create an IAM OIDC provider for the cluster:

eksctl utils associate-iam-oidc-provider --profile=perp \
--region ap-northeast-1 \
--cluster perp-staging \
--approve

aws iam list-open-id-connect-providers --profile=perp

ref:
https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html

Create an IAM policy that allows the role to access all secrets we created on AWS Secret Manager:

AWS_ACCOUNT_ID=$(aws sts get-caller-identity --profile=perp --query "Account" --output text)

cat <<EOF > policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecretVersionIds"
      ],
      "Resource": [
        "arn:aws:secretsmanager:ap-northeast-1:${AWS_ACCOUNT_ID}:secret:*"
      ]
    }
  ]
}
EOF

aws iam create-policy --profile=perp \
--policy-name perp-staging-secrets-policy --policy-document file://policy.json

Attach the above IAM policy to an IAM role, and define AssumeRole for the service account external-secrets-kubernetes-external-secrets which will be created later:

AWS_ACCOUNT_ID=$(aws sts get-caller-identity --profile=perp --query "Account" --output text)
OIDC_PROVIDER=$(aws eks describe-cluster --profile=perp --name perp-staging --region ap-northeast-1 --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///") 

cat <<EOF > trust.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com",
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:default:external-secrets-kubernetes-external-secrets"
        }
      }
    }
  ]
}
EOF

aws iam create-role --profile=perp \
--role-name perp-staging-secrets-role \
--assume-role-policy-document file://trust.json

aws iam attach-role-policy --profile=perp \
--role-name perp-staging-secrets-role \
--policy-arn YOUR_POLICY_ARN

ref:
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
https://gist.github.com/lukaszbudnik/f1f42bd5a57430e3c25034200ba44c2e

Deploy kubernetes-external-secrets Controller

helm repo add external-secrets https://external-secrets.github.io/kubernetes-external-secrets/

helm install external-secrets \
external-secrets/kubernetes-external-secrets \
--skip-crds \
--set env.AWS_REGION=ap-northeast-1 \
--set securityContext.fsGroup=65534 \
--set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"='YOUR_ROLE_ARN'

helm list

It would automatically create a service account named external-secrets-kubernetes-external-secrets in Kubernetes.

ref:
https://github.com/external-secrets/kubernetes-external-secrets/tree/master/charts/kubernetes-external-secrets

Deploy ExternalSecret

ExternalSecret app-secrets will generate a Secret object with the same name, and the content would look like:

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: example-secret
spec:
  backendType: secretsManager
  region: ap-northeast-1
  dataFrom:
    - YOUR_SECRET
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
    spec:
      containers:
      - name: example-app
        image: busybox:latest
        envFrom:
        - secretRef:
            name: example-secret

kubectl get secret example-secret -o jsonpath="{.data.KEY_1}" | base64 --decode

ref:
https://gist.github.com/lukaszbudnik/f1f42bd5a57430e3c25034200ba44c2e

Amazon Elastic Kubernetes Service (EKS) is a managed Kubernetes as a Service on AWS === Google Kubernetes Engine (GKE) on Google Cloud.

ref:
https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html
https://vinta.ws/code/the-complete-guide-to-google-kubernetes-engine-gke.html

Install CLI Tools

You need to install some command-line tools. Moreover, it would be better that you use the same Kubernetes version between client and server. Otherwise, you will get something like: WARNING: version difference between client (1.31) and server (1.33) exceeds the supported minor version skew of +/-1.

# kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/$(arch)/kubectl"
curl -LO "https://dl.k8s.io/release/v1.32.7/bin/darwin/arm64/kubectl"
sudo install -m 0755 kubectl /usr/local/bin

# eksctl
curl -LO "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_$(arch).tar.gz"
tar -xzf "eksctl_$(uname -s)_$(arch).tar.gz"
sudo install -m 0755 eksctl /usr/local/bin

# awscli
curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /

ref:
https://kubernetes.io/docs/tasks/tools/install-kubectl-macos/
https://github.com/eksctl-io/eksctl
https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

The following tools are also recommended which provide fancy terminal UIs to interact with your Kubernetes clusters.

# k9s
brew install derailed/k9s/k9s

# fubectl
curl -LO https://rawgit.com/kubermatic/fubectl/master/fubectl.source
source <path-to>/fubectl.source

ref:
https://github.com/derailed/k9s
https://github.com/kubermatic/fubectl

Create EKS Cluster

Use ClusterConfig to define the cluster.

# https://eksctl.io/usage/schema/
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: your-cluster
  region: ap-northeast-1
  version: "1.33"

# https://eksctl.io/usage/fargate-support/
fargateProfiles:
  - name: fp-default
    selectors:
      # all workloads in "fargate" Kubernetes namespace will be scheduled onto Fargate
      - namespace: fargate

# https://eksctl.io/usage/nodegroup-managed/
managedNodeGroups:
  - name: mng-m5-xlarge
    instanceType: m5.xlarge
    spot: true
    minSize: 1
    maxSize: 10
    desiredCapacity: 2
    volumeSize: 100
    nodeRepairConfig:
      enabled: true
    iam:
      withAddonPolicies:
        autoScaler: true
        awsLoadBalancerController: true
        cloudWatch: true
        ebs: true
      attachPolicyARNs:
        # default node policies (required when explicitly setting attachPolicyARNs)
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly
        # custom policies
        - arn:aws:iam::xxx:policy/your-custom-policy
    labels:
      service-node: "true"

# https://eksctl.io/usage/cloudwatch-cluster-logging/
cloudWatch:
  clusterLogging:
    enableTypes: ["api", "audit", "authenticator"]
    logRetentionInDays: 30

# https://eksctl.io/usage/addons/
addons:
  # networking addons are enabled by default
  # - name: kube-proxy
  # - name: coredns
  # - name: vpc-cni
  - name: amazon-cloudwatch-observability
  - name: aws-ebs-csi-driver
  - name: eks-pod-identity-agent
  - name: metrics-server
addonsConfig:
  autoApplyPodIdentityAssociations: true

# https://eksctl.io/usage/iamserviceaccounts/
iam:
  # not all addons support Pod Identity Association, so you probably still need OIDC
  withOIDC: true

# https://eksctl.io/usage/kms-encryption/
secretsEncryption:
  keyARN: arn:aws:kms:YOUR_ARN

# preview
AWS_PROFILE=perp eksctl create cluster -f cluster-config.yaml --dry-run

# create
eksctl create cluster -f cluster-config.yaml --profile=perp

If a nodegroup includes attachPolicyARNs, it must also include the default node policies, like AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy and AmazonEC2ContainerRegistryPullOnly.

ref:
https://eksctl.io/usage/schema/
https://eksctl.io/usage/iam-policies/
https://github.com/weaveworks/eksctl/tree/main/examples

Managed Nodegroups

You could re-use ClusterConfig to create more managed nodegroups after cluster creation. However, you can only create: most fields under managedNodeGroups section cannot be changed once created. If you need to tweak something, just add a new one.

Also, if you're not familiar with instance types on AWS, try Instance Selector which you only need to specify how much CPU, memory and GPU you want.

managedNodeGroups:
  - name: mng-spot-2c4g
    instanceSelector:
      vCPUs: 2
      memory: 4GiB
      gpus: 0
    spot: true
    minSize: 1
    maxSize: 5
    desiredCapacity: 2
    volumeSize: 100
    nodeRepairConfig:
      enabled: true

eksctl create nodegroup -f cluster-config.yaml --profile=perp

ref:
https://eksctl.io/usage/nodegroup-managed/
https://eksctl.io/usage/instance-selector/

Addons

When a cluster is created, EKS automatically installs vpc-cni, coredns and kube-proxy as self-managed addons. Here are other common addons you probably want to install as well:

• eks-pod-identity-agent: Manages IAM permissions for pods using Pod Identity Associations instead of OIDC
• amazon-cloudwatch-observability: Collects and sends container metrics, logs, and traces to CloudWatch Container Insights (Logs Insights)
• aws-ebs-csi-driver: Enables pods to use EBS volumes for persistent storage through Kubernetes PersistentVolumes
• metrics-server: Provides resource metrics (CPU/memory) for HPA, VPA and kubectl top commands

If not specified explicitly, addons will be created with a role that has all recommended policies attached.

It's worth noting that EKS Pod Identity Associations is AWS's newer, simpler way for Kubernetes pods to assume IAM roles, replacing the older IRSA (IAM Roles for Service Accounts) method.

ref:
https://eksctl.io/usage/addons/
https://eksctl.io/usage/pod-identity-associations/
https://eksctl.io/usage/iamserviceaccounts/

IAM Permissions

Pods run on regular nodes will behave as NodeInstanceRole; pods run on Fargate nodes will use FargatePodExecutionRole. If you would like to adjust IAM permissions for a nodegroup, use the following command to find out which IAM role it's using:

aws eks describe-nodegroup \
    --region ap-northeast-1 \
    --cluster-name your-cluster \
    --nodegroup-name mng-m5-xlarge \
    --query 'nodegroup.nodeRole' \
    --output text \
    --profile=perp

ref:
https://docs.aws.amazon.com/eks/latest/userguide/create-node-role.html

Access EKS Cluster

Instead of using the old aws-auth ConfigMap, you could use Access Entries to manage users/roles between AWS IAM and Kubernetes. Though, the user who running eksctl create cluster should remove him/her self from the access entries, EKS will add the user automatically.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: your-cluster
  region: ap-northeast-1

accessConfig:
  authenticationMode: API
  accessEntries:
    # - principalARN: arn:aws:iam::xxx:user/user1
    #   accessPolicies:
    #     - policyARN: arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy
    #       accessScope:
    #         type: cluster
    - principalARN: arn:aws:iam::xxx:user/user2
      accessPolicies:
        - policyARN: arn:aws:eks::aws:cluster-access-policy/AmazonEKSAdminPolicy
          accessScope:
            type: cluster
    - principalARN: arn:aws:iam::xxx:user/user3
      accessPolicies:
        - policyARN: arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy
          accessScope:
            type: cluster
    - principalARN: arn:aws:iam::xxx:user/production-eks-deploy
      type: STANDARD

# list
eksctl get accessentry --region ap-northeast-1 --cluster your-cluster --profile perp

# create
eksctl create accessentry -f access-entries.yaml --profile perp

ref:
https://eksctl.io/usage/access-entries/
https://docs.aws.amazon.com/eks/latest/userguide/access-policies.html

Connect kubectl to your EKS cluster by creating a kubeconfig file.

# setup context for a cluster
aws eks update-kubeconfig --region ap-northeast-1 --name your-cluster --alias your-cluster --profile=perp

# switch cluster context
kubectl config use-context your-cluster

kubectl get nodes

cat ~/.kube/config

ref:
https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html

Upgrade EKS Cluster

After upgrading your EKS cluster on AWS Management Console, you might find kube-proxy is still using the old image from previous Kubernetes - just upgrade them manually. Or if you are going to create an ARM-based nodegroup, you might also need to upgrade core addons first:

eksctl utils update-aws-node --region ap-northeast-1 --cluster your-cluster --approve --profile=perp
eksctl utils update-coredns --region ap-northeast-1 --cluster your-cluster --approve --profile=perp
eksctl utils update-kube-proxy --region ap-northeast-1 --cluster your-cluster --approve --profile=perp

kubectl get pods -n kube-system

If you got Error: ImagePullBackOff, try the following commands and replace the version (and region) with the version listed in Latest available kube-proxy container image version for each Amazon EKS cluster version:

kubectl describe daemonset kube-proxy -n kube-system | grep Image
kubectl set image daemonset.apps/kube-proxy -n kube-system \
kube-proxy=602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/kube-proxy:v1.25.16-minimal-eksbuild.8

ref:
https://eksctl.io/usage/cluster-upgrade/
https://github.com/weaveworks/eksctl/issues/1088

Delete EKS Cluster

You might need to manually delete/detach following resources in order to delete the cluster:

• Detach non-default policies from NodeInstanceRole and FargatePodExecutionRole
• Fargate Profile
• EC2 Network Interfaces
• EC2 Load Balancer

eksctl delete cluster --region ap-northeast-1 --name your-cluster --disable-nodegroup-eviction --profile=perp