Amazon EKS: Setup kubernetes-external-secrets with AWS Secret Manager

Amazon EKS: Setup kubernetes-external-secrets with AWS Secret Manager

kubernetes-external-secrets allows you to use external secret management systems, like AWS Secrets Manager, to securely add Secrets in Kubernetes, so Pods can access Secrets normally.

ref:
https://github.com/external-secrets/kubernetes-external-secrets

AWS Secret Manager

For instance, we create a secret named YOUR_SECRET on AWS Secret Manager in the same region as our EKS cluster, using DefaultEncryptionKey as the encryption key. The content of the secret entity look like:

{
  "KEY_1": "VALUE_1",
  "KEY_2": "VALUE_2",
}

We can retrieve the secret value:

aws secretsmanager get-secret-value --profile=perp \
--region ap-northeast-1 \
--secret-id YOUR_SECRET

kubernetes-external-secrets

For kubernetes-external-secrets to work properly, it must be granted access to AWS Secrets Manager. To achieve that, we need to create an IAM role for kubernetes-external-secrets' service account.

ref:
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

Configure Secrets Backends

Create an IAM OIDC provider for the cluster:

eksctl utils associate-iam-oidc-provider --profile=perp \
--region ap-northeast-1 \
--cluster perp-staging \
--approve

aws iam list-open-id-connect-providers --profile=perp

ref:
https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html

Create an IAM policy that allows the role to access all secrets we created on AWS Secret Manager:

AWS_ACCOUNT_ID=$(aws sts get-caller-identity --profile=perp --query "Account" --output text)

cat <<EOF > policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecretVersionIds"
      ],
      "Resource": [
        "arn:aws:secretsmanager:ap-northeast-1:${AWS_ACCOUNT_ID}:secret:*"
      ]
    }
  ]
}
EOF

aws iam create-policy --profile=perp \
--policy-name perp-staging-secrets-policy --policy-document file://policy.json

Attach the above IAM policy to an IAM role, and define AssumeRole for the service account external-secrets-kubernetes-external-secrets which will be created later:

AWS_ACCOUNT_ID=$(aws sts get-caller-identity --profile=perp --query "Account" --output text)
OIDC_PROVIDER=$(aws eks describe-cluster --profile=perp --name perp-staging --region ap-northeast-1 --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///") 

cat <<EOF > trust.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com",
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:default:external-secrets-kubernetes-external-secrets"
        }
      }
    }
  ]
}
EOF

aws iam create-role --profile=perp \
--role-name perp-staging-secrets-role \
--assume-role-policy-document file://trust.json

aws iam attach-role-policy --profile=perp \
--role-name perp-staging-secrets-role \
--policy-arn YOUR_POLICY_ARN

ref:
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
https://gist.github.com/lukaszbudnik/f1f42bd5a57430e3c25034200ba44c2e

Deploy kubernetes-external-secrets Controller

helm repo add external-secrets https://external-secrets.github.io/kubernetes-external-secrets/

helm install external-secrets \
external-secrets/kubernetes-external-secrets \
--skip-crds \
--set env.AWS_REGION=ap-northeast-1 \
--set securityContext.fsGroup=65534 \
--set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"='YOUR_ROLE_ARN'

helm list

It would automatically create a service account named external-secrets-kubernetes-external-secrets in Kubernetes.

ref:
https://github.com/external-secrets/kubernetes-external-secrets/tree/master/charts/kubernetes-external-secrets

Deploy ExternalSecret

ExternalSecret app-secrets will generate a Secret object with the same name, and the content would look like:

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: example-secret
spec:
  backendType: secretsManager
  region: ap-northeast-1
  dataFrom:
    - YOUR_SECRET
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
    spec:
      containers:
      - name: example-app
        image: busybox:latest
        envFrom:
        - secretRef:
            name: example-secret
kubectl get secret example-secret -o jsonpath="{.data.KEY_1}" | base64 --decode

ref:
https://gist.github.com/lukaszbudnik/f1f42bd5a57430e3c25034200ba44c2e

Amazon EKS: Create a Kubernetes cluster via ClusterConfig

Amazon EKS: Create a Kubernetes cluster via ClusterConfig

Amazon Elastic Kubernetes Service (Amazon EKS) is a managed Kubernetes as a Service on AWS. IMAO, Google Cloud's GKE is still the best choice of managed Kubernetes service if you're not stuck in AWS.

ref:
https://docs.aws.amazon.com/eks/latest/userguide/eks-ug.pdf
https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html

Also see:
https://vinta.ws/code/the-complete-guide-to-google-kubernetes-engine-gke.html

Installation

We need to install some command-line tools: aws, eksctl and kubectl.

brew tap weaveworks/tap
brew install awscli weaveworks/tap/eksctl kubernetes-cli

k9s and fubectl are also recommended which provides fancy terminal UIs to interact with your Kubernetes clusters.

brew install k9s

curl -LO https://rawgit.com/kubermatic/fubectl/master/fubectl.source
source <path-to>/fubectl.source

ref:
https://github.com/derailed/k9s
https://github.com/kubermatic/fubectl

Create Cluster

We use a ClusterConfig to define our cluster.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: perp-staging
  region: ap-northeast-1
# All workloads in the "fargate" Kubernetes namespace will be scheduled onto Fargate
fargateProfiles:
  - name: fp-default
    selectors:
      - namespace: fargate
# https://eksctl.io/usage/schema/
managedNodeGroups:
  - name: managed-ng-m5-4xlarge
    instanceType: m5.4xlarge
    instancePrefix: m5-4xlarge
    minSize: 1
    maxSize: 5
    desiredCapacity: 3
    volumeSize: 100
    iam:
      withAddonPolicies:
        cloudWatch: true
        albIngress: true
        ebs: true
        efs: true
# Enable envelope encryption for Kubernetes Secrets
secretsEncryption:
  keyARN: "arn:aws:kms:YOUR_KMS_ARN"
# Enable CloudWatch logging for Kubernetes components
cloudWatch:
  clusterLogging:
    enableTypes: ["*"]

Create the cluster.

eksctl create cluster --profile=perp -f clusterconfig.yaml

ref:
https://eksctl.io/usage/creating-and-managing-clusters/
https://github.com/weaveworks/eksctl/tree/master/examples

We can also use the same config file to update our cluster, but not all configurations are supported currently.

eksctl upgrade cluster --profile=perp -f clusterconfig.yaml

Access Cluster

aws eks --profile=perp update-kubeconfig \
--region ap-northeast-1 \
--name perp-staging \
--alias [email protected]

ref:
https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html

Delete Cluster

# You might need to manually delete/detach following resources first:
# Detach non-default policies for FargatePodExecutionRole and NodeInstanceRole
# Fargate Profile
# EC2 Network Interfaces 
# EC2 ALB
eksctl delete cluster --profile=perp \
--region ap-northeast-1 \
--name perp-staging

Then you can delete the CloudFormation stack on AWS Management Console.

Cluster Authentication

kubectl get configmap aws-auth -n kube-system -o yaml

We must copy mapRoles from the above ConfigMap, and add the mapUsers section:

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  # NOTE: mapRoles are copied from "kubectl get configmap aws-auth -n kube-system -o yaml"
  mapRoles: |
    - rolearn: YOUR_ARN_FargatePodExecutionRole
      username: system:node:{{SessionName}}
      groups:
      - system:bootstrappers
      - system:nodes
      - system:node-proxier
    - rolearn: YOUR_ARN_NodeInstanceRole
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
  # Only IAM users listed here can access this cluster
  mapUsers: |
    - userarn: YOUR_USER_ARN
      username: YOUR_AWS_USERNAME
      groups:
        - system:masters
kubectl apply -f aws-auth.yaml
kubectl describe configmap -n kube-system aws-auth

ref:
https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html
https://aws.amazon.com/premiumsupport/knowledge-center/amazon-eks-cluster-access/

Setup Container Logging for Fargate Nodes

Create an IAM policy and attach the IAM policy to the pod execution role specified for your Fargate profile. The --role-name should be the name of FargatePodExecutionRole, you can find it under "Resources" tab in the CloudFormation stack of your EKS cluster.

curl -so permissions.json https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json

aws iam create-policy --profile=perp \
--policy-name eks-fargate-logging-policy \
--policy-document file://permissions.json

aws iam attach-role-policy --profile=perp \
--policy-arn arn:aws:iam::XXX:policy/eks-fargate-logging-policy \
--role-name eksctl-perp-staging-cluste-FargatePodExecutionRole-XXX

Configure Kubernetes to send container logs on Fargate nodes to CloudWatch via Fluent Bit.

kind: Namespace
apiVersion: v1
metadata:
  name: aws-observability
  labels:
    aws-observability: enabled
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match *
        region ap-northeast-1
        log_group_name /aws/eks/perp-staging/containers
        log_stream_prefix fluent-bit-
        auto_create_group On
kubectl apply -f aws-logs-fargate.yaml

ref:
https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html
https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch

Setup Container Logging for EC2 Nodes (CloudWatch Container Insights)

Deploy Fluent Bit as DaemonSet to send container logs to CloudWatch Logs.

ClusterName=perp-staging
RegionName=ap-northeast-1
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
curl -s https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/;s/{{region_name}}/'${RegionName}'/;s/{{http_server_toggle}}/"'${FluentBitHttpServer}'"/;s/{{http_server_port}}/"'${FluentBitHttpPort}'"/;s/{{read_from_head}}/"'${FluentBitReadFromHead}'"/;s/{{read_from_tail}}/"'${FluentBitReadFromTail}'"/' > aws-logs-ec2.yaml
kubectl apply -f aws-logs-ec2.yaml

ref:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-EKS-quickstart.html

How HTTPS Works in Layman's Terms - TLS 1.2 and 1.3

How HTTPS Works in Layman's Terms - TLS 1.2 and 1.3

HTTPS stands for Hypertext Transfer Protocol Secure, also commonly referred to as HTTP over TLS or HTTP over SSL. HTTPS is not a separate protocol from HTTP, it merely means using SSL/TLS to encrypt an HTTP request and response.

ref:
https://howhttps.works/

SSL/TLS

Basically, SSL (Secure Sockets Layer) and TLS (Transport Layer Security) is the same thing. TLS is the modern version of now-deprecated SSL. In most contexts, both terms are exchangeable.

TLS is a security protocol which mainly performs three tasks:

  • Privacy - encrypting data between client and server using Encryption Algorithms.
  • Authentication - ensuring that server is who it claims to be using Certificates.
  • Integrity - verifying that data have not been forged using Message Authentication Code (MAC).

In addition to the use case of HTTPS, TLS can also be used to encrypt other communications such as Email or VoIP.

ref:
https://www.cloudflare.com/learning/ssl/transport-layer-security-tls/

Encryption Algorithms

There are 2 types of encryption algorithms:

  • Symmetric Encryption
    • There is only one key: the client and server use the same key to encrypt and decrypt.
    • Fast and cheap (nanoseconds per operation).
    • A common algorithm is AES.
  • Asymmetric Encryption (also known as Public-key Encryption)
    • There is a pair of two keys: the public key encrypts the message, and only the corresponding private key can decrypt it.
    • Slow and expensive (microseconds to milliseconds per operation).
    • Some common algorithms are RSA and Diffie-Hellman (DH).

TLS actually uses both Asymmetric Encryption and Symmetric Encryption, so-called a hybrid cryptosystem. Simply speaking, TLS first uses an asymmetric algorithm to exchange shared secrets between both sides, then generates a symmetric key (the session key) from the shared secrets, finally uses the session key to encrypt application data (HTTP request/response). A cryptographic system involves certificates and public-key encryption is often called Public Key Infrastructure (PKI).

I'm not an expert in Cryptography or Information Security, but I'm going to talk about what RSA and Diffie-Hellman are a little bit since they are crucial to TLS.

ref:
https://www.thesslstore.com/blog/types-of-encryption-encryption-algorithms-how-to-choose-the-right-one/

RSA

There is a pair of two keys in RSA, the public key can be shared publicly, and the private key, as the name suggests, must be kept secret. Data encrypted with the public key can only be decrypted with the private key, and vice versa. Since RSA is a relatively slow algorithm, we usually generate a key pair and use them for every connection. So RSA keys are considered static.

RSA is also often used for Digital Signature. In this case, the message is hashed first since RSA operations can't handle messages longer than the key size. The sender generates a signature by signing (encrypting) the hash with its own private key, and sends both the message and the signature to the receiver. The receiver also hashes the message first, and verifies (decrypts) the signature with the corresponding public key, and checks whether the decrypting hash equals the hash of the message. If they are equal, the message is indeed sent by the sender because no one else has the private key.

ref:
https://www.comparitech.com/blog/information-security/digital-signatures/

Diffie-Hellman (DH)

There are many variants of Diffie-Hellman, for instance, Diffie-Hellman Ephemeral (DHE), Elliptic Curve Diffie–Hellman (ECDH), and Elliptic Curve Diffie-Hellman Ephemeral (ECDHE).

Let's talk about how DHE works first:

  1. Both client and server agree on a set of DH parameters: g (generator) and p (prime).
    • Instead of exchange, g and p are usually predefined in the software that client and server use.
    • These values are public so it's ok that an attacker knows them.
  2. Each of client and server generates a random number as the private key, and calculates the public key from DH formula 1: (g^own_private_key) mod p.
    • There are 2 key pairs:
      • Client private key
      • Client public key
      • Server private key
      • Server public key
    • Since the 2 private keys are randomly generated for every connection, this is the "Ephemeral" part of DHE.
  3. Each of client and server sends their public keys to the other side.
  4. Each of client and server calculates the same shared secret from DH formula 2: (the_other's_public_key^own_private_key) mod p.
    • Client's shared secret = (server_public_key^client_private_key) mod p.
    • Server's shared secret = (client_public_key^server_private_key) mod p.
    • Magically, client's shared secret == server's shared secret.

Then we can use the shared secret for Symmetric Encryption.

In step 2, if both client and server always use the same private keys for every connection, that is Static DH. Because the key pairs are temporary, a compromise of private keys does not jeopardize the privacy of other DH connections. This is known as Perfect Forward Secrecy (PFS). Moreover, if we replace the DH formula in step 2 and 4 with an elliptic curve formula, that is ECDH.

ref:
https://www.wst.space/ssl-part-2-diffie-hellman-key-exchange/
https://crypto.stackexchange.com/questions/67797/in-diffie-hellman-are-g-and-p-universal-constants-or-are-they-chosen-by-one

Certificates

To obtain a valid SSL certificate, the server first needs to create a Certificate Signing Request (CSR) file with an RSA private key and submits it to a Certificate Authority (CA). A CA is an organization, a trusted third-party that generates and gives out SSL certificates. The CA will also sign the certificate with its private key, allowing clients to verify it with CA's public key. Operating systems and browsers have pre-installed public keys of all of the major CAs.

A SSL certificate contains:

  • The domain name and associated subdomains
  • The issuer (CA)
  • The CA's digital signature
  • The expiration date of the certificate
  • The server's public key

A certificate is actually a chain of multiple certificates (Chain of Trust), usually three or more: the server's certificate, the intermediate CA's certificate, and the root CA's certificate. In a TLS communication, the client first verifies the signature of the server's certificate with the intermediate CA's public key, and checks the signature of the intermediate CA's certificate with the root CA's public key. Finally, the root CA's certificate is inherently trusted by OSs or browsers. If any verification fails or the root certificate is not trusted (this would be a self-signing certificate), the TLS communication terminates.

Clients also checks if the certificate is for the correct domain.

ref:
https://www.cloudflare.com/learning/ssl/what-is-an-ssl-certificate/
https://security.stackexchange.com/questions/56389/ssl-certificate-framework-101-how-does-the-browser-actually-verify-the-validity

Message Authentication Code (MAC) Algorithms

Simply speaking, the sender calculates a MAC code by doing mac_code = mac_function(key, message), then sends the message along with the MAC code to the receiver. The receiver also calculates a MAC code of the message using the same MAC algorithm and the shared secret key, and checks whether both MAC values are equal. If they are equal, the integrity of the message is confirmed. A MAC code is sometimes called a checksum.

A common MAC algorithm is HMAC.

ref:
https://crypto.stackexchange.com/questions/5646/what-are-the-differences-between-a-digital-signature-a-mac-and-a-hash

TLS Handshake

TLS handshake is the foundational part of a HTTPS communication which happens after establishing TCP connection and before the HTTP request/response cycle. The purpose of TLS handshake is to negotiate a session key used to encrypt/decrypt HTTP data.

Most of major browsers have already dropped support for TLS 1.0 and 1.1 in 2020, also, TLS 1.2 and 1.3 are currently the most widely used versions of TLS. We will focus on later ones. Since TLS 1.2 supports multiple algorithms for key exchange but TLS 1.3 only uses Diffie-Hellman (RSA has been completely removed), so we are going to talk about TLS handshake with RSA in TLS 1.2 and TLS handshake with Diffie-Hellman in TLS 1.3.

ref:
https://www.cloudflare.com/learning/ssl/what-happens-in-a-tls-handshake/
https://www.thesslstore.com/blog/explaining-ssl-handshake/
https://www.thesslstore.com/blog/cipher-suites-algorithms-security-settings/

TLS Handshake with RSA in TLS 1.2

Before the TLS handshake, both client and server need to establish a TCP connection if HTTP/1.1 or HTTP/2 is using (HTTP/3 would be another story). TCP is bi-directional, and there could be multiple packets sent in one trip.

TLS handshake in TLS 1.2 takes 2 roundtrips:

| Client                                            | Server                                            |
|---------------------------------------------------|---------------------------------------------------|
| -> TCP: SYN                                       |                                                   |
|                                                   | <- TCP: SYN ACK                                   |
| -> TCP: ACK                                       |                                                   |
| -> TLS: Client Hello (plaintext)                  |                                                   |
|                                                   | <- TLS: Server Hello (plaintext)                  |
|                                                   | <- TLS: Server Certificate (plaintext)            |
|                                                   | <- TLS: Server Hello Done (plaintext)             |
| -> TLS: Client Key Exchange (plaintext)           |                                                   |
| -> TLS: Client Change Cipher Spec (plaintext)     |                                                   |
| -> TLS: Client Handshake Finished (encrypted)     |                                                   |
|                                                   | <- TLS: Server Change Cipher Spec (plaintext)     |
|                                                   | <- TLS: Server Handshake Finished (encrypted)     |
| -> HTTP: Request (encrypted)                      |                                                   |
|                                                   | <- HTTP: Response (encrypted)                     |
  • Client Hello
    • Sending following data:
      • A random "Client Random" string.
      • A list of supported Cipher Suites (a list of cryptographic algorithms).
  • Server Hello
    • Sending following data:
      • A random "Server Random" string.
      • The selected Cipher Suite.
  • Server Certificate
    • Sending SSL certificate contains server's public key.
  • Server Hello Done
    • Telling client that it has sent over above messages.
  • Client Key Exchange
    • Sending a random "Pre-master Secret" encrypted with server's public key.
  • Client Change Cipher Spec
    • Generating the session key from Client Random, Server Random, and Pre-master Secret.
    • Telling server that it's ready for encrypted communication.
  • Client Finished
    • Sending a verification data encrypted with the session key.
  • Server Change Cipher Spec
    • Generating the session key from Client Random, Server Random, and Pre-master Secret.
    • Telling client that it's ready for encrypted communication.
  • Server Finished
    • Sending a verification data encrypted with the session key.

After the TLS handshake, both client and server start to encrypt/decrypt application data with the session key.

For a complete byte-by-byte illustration of TLS 1.2 Handshake, see:
https://tls.ulfheim.net/

ref:
https://medium.com/kuranda-labs-engineering/tls-6d9f75adba9f

TLS Handshake with Elliptic Curve Diffie-Hellman Ephemeral (ECDHE) in TLS 1.3

TLS or any encrypted communications have always added overhead when it comes to performance. One of the significant changes of TLS 1.3 is that the TLS handshake in TLS 1.3 only requires one roundtrip instead of 2, which makes TLS 1.3 much faster than older versions.

TLS 1.3 has also reduced the number of supported Cipher Suites from 37 to 5 by removing weak and less-used cryptographic algorithms, and only supports ECDHE for key exchange algorithms, which means that client can send its key share information right away at the beginning of the handshake. In other words, TLS 1.3 merges Client Key Exchange into Client Hello.

TLS handshake in TLS 1.3 takes 1 (and a half actually) roundtrip:

| Client                                            | Server                                            |
|---------------------------------------------------|---------------------------------------------------|
| -> TCP: SYN                                       |                                                   |
|                                                   | <- TCP: SYN ACK                                   |
| -> TCP: ACK                                       |                                                   |
| -> TLS: Client Hello (plaintext)                  |                                                   |
|                                                   | <- TLS: Server Hello (plaintext)                  |
|                                                   | <- TLS: Wrapper (encrypted)                       |
|                                                   |         Server Certificate                        |
|                                                   |         Server Handshake Finished                 |
| -> TLS: Wrapper (encrypted)                       |                                                   |
|         Client Handshake Finished                 |                                                   |
| -> HTTP: Request (encrypted)                      |                                                   |
|                                                   | <- HTTP: Response (encrypted)                     |
  • Client Hello
    • Before Client Hello, calculating an ephemeral key pair for ECDHE key share based on the selected curve.
    • Sending following data:
      • A random "Client Random" string.
      • A list of supported Cipher Suites.
      • The selected curve (DH parameters: g and p) for ECDHE.
      • The client's ECDHE public key.
  • Server Hello
    • Before Server Hello, calculating an ephemeral key pair for ECDHE key share based on client's selected curve.
    • Sending following data:
      • A random "Server Random" string.
      • The selected Cipher Suite.
      • The server's ECDHE public key.
  • Server Wrapper
    • Before Server Wrapper, generating the handshake key from its own ECDHE private key and client's ECDHE public key.
    • Containing following records:
      • Server Certificate
      • Server Handshake Finished
      • They are encrypted with the handshake key.
    • After Server Wrapper, generating the session key from Client Random, Server Random, and the handshake key.
  • Client Wrapper
    • Before Client Wrapper, generating the handshake key from its own ECDHE private key and server's ECDHE public key.
    • Containing following records:
      • Client Handshake Finished
      • They are encrypted with the handshake key.
    • After Client Wrapper, generating the session key from Client Random, Server Random, and the handshake key.

After the TLS handshake, both client and server start to encrypt/decrypt application data with the session key.

For a complete byte-by-byte illustration of TLS 1.3 Handshake, see:
https://tls13.ulfheim.net/

ref:
https://www.thesslstore.com/blog/tls-1-3-everything-possibly-needed-know/