• should I use a User Pool or an Identity Pool?
• If I create a User Pool, do I need to use a federated Identity Provider?
• When the documentation says that Cognito can be used as an OIDC what does it mean?

The goal of this article is to shed some light on this topics and help you set Cognito for your project. We’ll use go for the examples, but should be able to understand the ideas behind the code an dapply them to a project using other languages.

Let’s start by discussing some basic topics, before we start creating terraform templates and writing go code.

AWS Cognito basic concepts

AWS Cognito provides a complete solution for authentication and authorization. Cognito can be used in different scenarios, for example:

• You need a complete solution to mange, sign-up, sign-in, etcetera, in your app
• Cognito can also serve as an identity provider that can return Oauth 2.0 acces tokens, this tokens contain metadata of the authenticated user so you can add authorization logic to your code based on this information
• You need access to AWS resources. With AWS Cognito you can return AWS credentials (Specified using IAM) to access those resources
• Cognito can also serve as an intermediate Service Provider to other Identity providers. For example, Facebook, Google, etc.

User Pools

User pools are, as the name suggests, a repository of users you want to keep track off. The user pool can be an independent directory, that means that all the users live in the User Pool. The user pool can also source the users from third-party providers. That means that the User user pool will be an intermediate service provider. Cognito user pools can be used as an OIDC (OpenID Connect) Identity Provider.

The simplest form is to use the user pool as an independent directory. Everything is handled inside AWS Cognito, which makes things simpler. When using the user pools as an intermidate SP to third parties, help translating all the external tokens returned by the third-party IdPs to Cognito Tokens, so everything is standardised into one format and this could simplify your code.

App Client

In Cognito creating the User Pool is only half the batlte, you still need to interact with it. While you can in certain cases just interact with the User Pool programatically using the AWS SDK, it is far easier to interact with the User Pool via an app client. The app client provides features like the following:

• Authentication Flows
• Token management. For example, token expiration.
• OAuth scope management
• Callback and logout URLs for the authentication workflow
• CORS configuration
• Security features, like Secrets.
• Hosted UI that you can customize
• Manage identity providers for the app client

In general you’ll need to set up an app client for your application to communicate with Cognito for user authentication, token retrieval and expirtation, etcetera.

Identity Pools

The identity pools are used in combination with authenticated users from User Pools, or it can use federated identities from external IdPs. When an entity has been authenticated, then we can provide AWS Credentials to access AWS resources. For example, we have an app that shows the content of a file in private S3 bucket. We can create an IAM role that grants access to the S3 bucket content. Instead of adding someone to your AWS organisation, you could use an Identity Pool to get temporary AWS credentials to assume that IAM role and be able to access the file form the private S3 bucket.

Identity Providers (IdPs)

A user pool can serve as an Identity Provider, meaning that it handles the verification of your users and can provide your app with information about them. When you use an external IdP, you can autenticate and get information about your users from a service that you don’t manage, it is also know as federated identities. You have probably seen sign in with Google, Facebook, Amazon, etcetera, they serve as Identity Providers. Their users are already registered in their service and their identity is being federated from them. You can also use an Active Directory as your identity provider, or any other service that can communicate using SAML, OAuth2.0 or OpenID Connect(OIDC).

Creating the user pool with terraform

First, we need to create our user pool. That is the easy part, the only required field is the name of the user pool:

1
2
3
4
5

resource "aws_cognito_user_pool" "user_pool" {
  name = var.user_pool_name
  
  alias_attributes = ["email"]
}

Adding the alias_attributes to include the email, means that the user would be able to login using its username or the email.

Now we need to set up our App client:

1
2
3
4
5

resource "aws_cognito_user_pool_client" "user_pool_client" {
  name            = var.user_pool_client_name
  user_pool_id    = aws_cognito_user_pool.user_pool.id
  generate_secret = var.generate_client_secret
}

We are not going to be using custom domains, so we just need a prefix for our domain:

1
2
3
4
5
6
7

resource "aws_cognito_user_pool_domain" "custom_domain" {
  domain       = var.custom_domain
  user_pool_id = aws_cognito_user_pool.user_pool.id
  # If we were to use a custom domain we would need the certificate to be managed
  # by AWS Certificate Manager, in us-east-1
  # certificate_arn = aws_acm_certificate.cert.arn
}

In this post, we’ll explore basic directory structures used for Terraform projects.

Note: If you want a more in-depth discussion about the state and directory structure relationship, you might like my guide Meditations on Directory Structure for Terraform Projects.

Common ways to structure your directories and files for Terraform projects

There are multiple ways you can structure your code, Terraform is very flexible about it, so there is no single answer to that question. But let’s look at a few commong patterns.

Per-project boundary

A common pattern is structuring your code using the environment as the boundary for separating resources. The directory structure looks like the following:

── basic_project
    ├── environments
    │   ├── dev
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   └── variables.tf
    │   ├── prod
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   └── variables.tf
    │   └── staging
    │       ├── main.tf
    │       ├── outputs.tf
    │       └── variables.tf
    ├── modules
    └── shared

This structure is clean and easy to understand at a glance. If you work in a specific environment, any change you make won’t affect other environments. It has a couple of drawbacks. For example, we could easily add different resources to each environment, creating drift between the environments. It is good practice to keep your environments as similar as possible, so running tests in a lower environment will reflect the effects in the same way as running them in a production environment.

In a clean directory structure, the code inside the environment directories (i.e. dev, staging and prod) are primarily to reference modules. The idea is to reduce code duplication and use the modules as the source of truth of how resources should be configured and with sensible default behaviours.

Per-service boundary

When the number of resources is high, some teams break down the directory structure into smaller areas of interest. For example, split the directory structure by service. The file structure would look like the following:

── service_A
    └── environments
        ├── dev
        │   ├── main.tf
        │   ├── outputs.tf
        │   └── variables.tf
        ├── prod
        │   ├── main.tf
        │   ├── outputs.tf
        │   └── variables.tf
        └── staging
            ├── main.tf
            ├── outputs.tf
            └── variables.tf

If you are observant, you’ll notice that the directory structure is basically the same as the previous one. Breaking down into smaller areas is similar to breaking it down by project. The idea stays the same: create a boundary to hold certain resources.

Depending on the number of resources, the per-project structure might start causing your plan and apply commands to take a lot of time. The reason is that Terraform has to query the AWS API to check the state of the resources it is tracking. So, a larger number of resources will lead to more API calls.

There are other reasons why we might need to segregate the state:

• High Number of resources, we already mentioned this
• Using different AWS accounts per environment
• Breaking state per region. Projects sometimes segregate their state per region to:
  • Reduce blast radius if a region is down
  • Deployment strategies that would prefer to only deploy to certain regions for testing, i.e. canary deployments

Using workspaces

Another common alternative for the per-project directory structure is using workspaces. This structure maintains the same code for all environments, reducing the possibility of differences between environments.

── single_code_multiple_var
    ├── infra
    │   ├── main.tf
    │   ├── provider.tf
    │   ├── outputs.tf
    │   └── variables.tf
    ├─── environments
    │   ├── env.dev.tfvars
    │   ├── env.staging.tfvars
    │   └── env.prod.tfvars
    └── modules

There are still cases where we would like different behaviours per environment. For example, using smaller instance types for lower-level environments and larger instance types for production. To reflect that difference, we use different values in tfvars files that will capture the differences. This directory structure and workflow are OK if you are OK with having the same backend for all the workspaces:

terraform {
  backend "s3" { <--- we can't use variable interpolation, so we can't do the following
    region         = "${local.env[var.enviroment]}"
    bucket         = "rderik-tfstate-${var.environment}"
    key            = "${var.environment}/tfstate/terraform.tfstate"
    dynamodb_table = "rderik-tfstate-${var.environment}"
    encrypt        = true
  }
}

If, for some reason, we need to use a different backend per environment, then we have to solve the problem with the backend, which has its own set of peculiarities.

Final Thoughts

The directory structure that we end up using for our project is highly dependent on how we want to manage the state of our project. Consider if you want to keep one state per project, service, region, etc. The key to deciding which directory structure to use in your project is for you to understand the relationship between:

• State
• Directory structure
• Backend
• Module versions

Once you clearly understand how you want to handle those aspects of your project, it’ll be easier to choose the directory structure that will make your life easier. But don’t worry too much about it. It is OK to start with one structure and later understand that you have outgrown that structure and migrate out of it. Believing you’ll have all the answers from the beginning is a common mistake.

Software projects are iterative. You keep growing and you keep adapting that is the nature of our work.

In this post, we are going to use the External Secrets Operator (ESO) to get the private SSH key from AWS SSM Parameter Store and inject it into ArgoCD using a Kubernetes Secret.

If you already have ArgoCD setup, skip directly to the section Setting up ArgoCD for private repositories. If not, follow along.

Ok, let’s get started.

Bash Beyond Basics Increase your efficiency and understanding of the shell

If you are interested in this topic you might enjoy my course Bash Byond Basics. This course helps you level up your bash skills. This is not a course on shell-scripting, is a course on improving your efficiency by showing you the features of bash that are seldom discussed and often ignored.

Every day you spend many hours working in the shell, every little improvement in your worklflows will pay dividends many fold!

Learn more

Installing ArgoCD

First, we need to install ArgoCD in our Kubernetes cluster. We will use Helm to do this, but we will do it in a declarative way so we can use our code as the source of truth. Let’s start by creating the directory and the required files for our Helm chart.

1
2

mkdir argo-cd
touch argo-cd/{Chart,requirements}.yaml

The content for the Chart.yaml file is:

1
2

name: argo-cd
version: 0.1.0

The content for requirements.yaml is:

1
2
3
4

dependencies:
  - name: argo-cd
    version: 5.13.6
    repository: https://argoproj.github.io/argo-helm

Now we can install the chart:

1
2
3

cd argo-cd/
helm dependency update
helm install argo-cd . -n argocd  --create-namespace

That should get ArgoCD set up in our Kubernetes Cluster. We need a repository with some Kubernetes templates. If you don’t have one, you can use the one I created for this post. It is a public repository. You can find it here. Fork it, and work with it. Later you can make it private or create a different private repository.

Create the ArgoCD Project and application

We will test that everything works correctly with ArgoCD using a public repository. Once ArgoCD runs correctly, we can focus on using a private repo. You can skip this section if you don’t want to run these tests. Let’s now create a new ArgoCD project and application.

The project:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: server-proj
  namespace: argocd
  # Finaliser that ensures that the project is not deleted until any application does not reference it
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  # Project description
  description: Server Project
  destinations:
    # Update your namespace
    - namespace: my-namespace
      server: https://kubernetes.default.svc
  sourceRepos:
    # Make sure to add your repository here
    - https://github.com/rderik/slartybartfast.git
  clusterResourceWhitelist:
  - group: ''
    kind: '*'

The application:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: server
  namespace: argocd
spec:
  destination:
    namespace: my-namespace
    server: https://kubernetes.default.svc
  project: server-proj
  source:
    # Change the repository to your repository
    repoURL: https://github.com/rderik/slartybartfast.git
    # point the path to the path where your Kubernetes templates are
    path: server
  syncPolicy:
    automated: # automated sync by default retries failed attempts 5 times with following delays between attempts ( 5s, 10s, 20s, 40s, 80s ); retry controlled using `retry` field.
      prune: false # Specifies if resources should be pruned during auto-syncing ( false by default ).
      selfHeal: false # Specifies if partial app sync should be executed when resources are changed only in target Kubernetes cluster and no git change detected ( false by default ).
      allowEmpty: true # Allows deleting all application resources during automatic syncing ( false by default ).

Ok, we can create the project and the application:

kubectl apply -f project.yaml
kubectl apply -f application.yaml

Finally, we can view our application in the ArgoCD dashboard by doing a port forwarding to the argocd-server service:

1

kubectl port-forward svc/argocd-server -n argocd 8080:443

I run most of my tests in a VM, so when I do port forwarding, I need the port to be bound to all interfaces:

1

kubectl port-forward svc/argo-cd-argocd-server -n argocd 8080:443 --address=0.0.0.0

We can now visit the ArgoCD dashboard at https://localhost:8080 and log in with the default username and password. The default username is admin. To obtain the default password, you need to read a secret created by default during installation.

1

kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d

Log in, and you should see your application being created!

If we do a port forward to the newly created pod, we can see the Nginx welcome page:

1

kubectl port-forward $(kubectl get pods -n my-namespace --no-headers | awk '{print $1}') 8081:80 -n my-namespace

In another terminal:

1

curl localhost:8081

Ok, now we have a working ArgoCD installation. We can focus on setting up the private repository access.

Setting up the private repository access

We need to create an SSH key pair. We will use the key pair to access the private repository.

1

$ ssh-keygen -t rsa -b 4096 -C "key for private repository" -f id_rsa_private_repo

Now we need to add the public key to the repository. In my case, I’m using GitHub, so I need to add the public key to the repository. If you also use GitHub, go to the repository Settings > Deploy Keys, and add the PUBLIC key. We only need read-only, so there is no need to add write permissions.

Let’s create a secure string parameter in AWS SSM parameter store that contains the PEM-encoded private key. We will use this as a secret to add the private key to ArgoCD.

1
2
3
4

aws ssm put-parameter \
  --name "/argo-cd/github-repo/ssh-key" \
  --type SecureString \
  --value "$(cat id_rsa_private_repo)"

Finally, now we can look at setting up ArgoCD for our private repository.

Setting up ArgoCD for private repositories

We need to create a secret that will contain the private key. We will use the private key to access the private repository.

ArgoCD’s documentation explains that we need to create a secret with the private key so ArgoCD can access the repository. The secret looks like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

apiVersion: v1
kind: Secret
metadata:
  name: private-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: git@github.com:argoproj/my-private-repository
  sshPrivateKey: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    ...
    -----END OPENSSH PRIVATE KEY-----

This approach has the issue that we would have to write our private key in plain text in the secret template. We don’t want to do that. So we have a few options to avoid that. We can create a template and documentation that explains how to fill up this file and then manually apply it. That works, but it is a little bit cumbersome. We’ll use External Secrets Operator and avoid doing that.

Setting up External Secrets Operator

As we did before for the other dependencies, we will set up the External Secrets Operator in a declarative way using Helm.

1
2

mkdir external-secrets-operator/
touch external-secrets-operator/{Chart,requirements}.yaml

The content for the Chart.yaml file is:

1
2

name: external-secrets-operator
version: 0.1.0

The content for requirements.yaml is:

1
2
3
4

dependencies:
  - name: external-secrets
    version: 0.6.1
    repository: https://charts.external-secrets.io

Now we can install the chart:

1
2
3

cd external-secrets-operator
helm dependency update
helm install external-secrets-operator . -n external-secrets-operator  --create-namespace

Perfect, we have everything set up. Now we can create the secret that will contain the private key.

Creating the secret

Make sure that your IAM role has access to the SSM parameter store. The role will need a policy similar to the following:

resource "aws_iam_policy" "ssm_parameter_policy" {
  name = "cluster-ssm-parameter-policy"

  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Action" : [
          "ssm:DescribeParameters",
          "ssm:GetParameter",
          "ssm:GetParameters",
          "ssm:GetParametersByPath"
        ],
        "Effect" : "Allow",
        "Resource" : [
          "arn:aws:ssm:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:/argo-cd/*"
        ]
      },
    ]
  })
}

With that out of the way, let’s define the Secret Store:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: my-aws-secret-store
  # we need the secret to exist in the same namespace as ArgoCD
  namespace: argocd
spec:
  provider:
    aws:
      service: ParameterStore
      # define a specific role to limit access
      # to certain secrets
      region: us-east-1

We are going to get the AWS credentials from the current environment. If you wish to use a secret and key stored in a Kubernetes secret, you could add the section:

      # Auth defines the information necessary to authenticate against AWS by
      # getting the accessKeyID and secretAccessKey from an already created Kubernetes Secret
      auth:
        secretRef:
          accessKeyID:
            name: awssm-secret
            key: access-key
          secretAccessKey:
            name: awssm-secret
            key: secret-access-key

Reference: https://external-secrets.io/v0.6.1/api/secretstore/

Ok, now we can create the external secret. If we take a look at the documentation, the secret should look like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

apiVersion: v1
kind: Secret
metadata:
  name: private-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: git@github.com:argoproj/my-private-repository
  sshPrivateKey: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    ...
    -----END OPENSSH PRIVATE KEY-----

To generate a secret with that structure, we will use the template feature of the External Secrets Operator to create the secret. The external secret will look like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: private-repo-ssh-key
  namespace: argocd
spec:
  # SecretStoreRef defines which SecretStore to use when fetching the secret data
  secretStoreRef:
    name: my-aws-secret-store
    kind: SecretStore  # or ClusterSecretStore
    # Specify a blueprint for the resulting Kind=Secret
  target:
    name: my-aws-secret-repo-ssh-key
    template:
      metadata:
        labels:
          argocd.argoproj.io/secret-type: repository
      # Use inline templates to construct your desired config file that contains your secret
      data:
        type: git
        url: git@github.com:rderik/slartypriv.git
        sshPrivateKey: |
          {{ .sshPrivateKey | toString }}
  data:
  - secretKey: sshPrivateKey
    remoteRef:
      key: /argo-cd/github-repo/ssh-key

Remember to update the url and your SSM parameter key to match your repositories. Remember you are going to be using ssh and not https.

With that out of the way, we can apply the two templates:

1
2

kubectl apply -f secret-store.yaml
kubectl apply -f external-secret.yaml

If we get any errors and need to review what is going on, we can use the following command:

1
2

kubectl describe SecretStore my-aws-secret-store -n argocd
kubectl describe ExternalSecret private-repo-ssh-key -n argocd

And if we want to check if the secret created in Kubernetes matches our SSH key:

1
2

kubectl get secret my-aws-secret-repo-ssh-key \
  -n argocd -o jsonpath="{.data.sshPrivateKey}" | base64 -d

That’s it. Now we can use our private repository in ArgoCD.

I created a private repository with the following url:

1

git@github.com:rderik/slartypriv.git

And the project in ArgoCD looks like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: server-proj-private
  namespace: argocd
  # Finaliser that ensures that the project is not deleted until any application does not reference it
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  # Project description
  description: Server Project
  destinations:
    # Update your namespace
    - namespace: my-namespace-private
      server: https://kubernetes.default.svc
  sourceRepos:
    # Make sure to add your repository here
    - git@github.com:rderik/slartypriv.git
  clusterResourceWhitelist:
  - group: ''
    kind: '*'

The application looks like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: server-private
  namespace: argocd
spec:
  destination:
    namespace: my-namespace-private
    server: https://kubernetes.default.svc
  project: server-proj-private
  source:
    # Change the repository to your repository
    repoURL: git@github.com:rderik/slartypriv.git
    # point the path to the path where your Kubernetes templates are
    path: server
  syncPolicy:
    automated:
      prune: false
      selfHeal: false
      allowEmpty: true

Notice that the sourceRepos and the repoURL have changed from https to ssh, so update your repository URLs correctly to use ssh. If we apply the project and the application, we should see it working without a problem in the ArgoCD dashboard.

1
2

kubectl apply -f project-private.yaml
kubectl apply -f application-private.yaml

And That’s it! We now have a private repository in ArgoCD.

Final thoughts

There are many moving parts when we are working with Kubernetes and ArgoCD, and we need to pay a lot of attention to all the small details. I got stuck in the past debugging for a long time until I realised that the repoURL was still using https instead of ssh. And I got stuck another time until I realised I had a typo on the Secret template.

Anyways, I hope this is useful.

References

• External Secrets Operator (ESO)
• ESO - AWS Parameter Store Provider
• ESO - ExternalSecret
• ESO - SecretStore
• ESO - advanced templating
• ArgoCD - declarative setup

“Implement the principle of least privilege and enforce separation of duties with appropriate authorization for each interaction with your AWS resources. Centralize identity management, and aim to eliminate reliance on long-term static credentials.”

Well Architected Framework - Security Pillar - Design principles

When we start using Kubernetes in AWS EKS, we might take some shortcuts during the learning phase and add all the policies to a role that we directly assign to our nodes. The issue is that we give much more privileges than we require just for practicality. And it can also become a habit. The way we learn something sometimes becomes our default pattern. So let’s break that habit and explore how to set IAM Roles to Kubernetes service accounts in AWS EKS.

Note: If you are interested in learning more about how to set up the directory structure for your Terraform project, you might find my guide, Meditations on Directory Structure for Terraform Projects, useful.

Bash Beyond Basics Increase your efficiency and understanding of the shell

If you are interested in this topic you might enjoy my course Bash Byond Basics. This course helps you level up your bash skills. This is not a course on shell-scripting, is a course on improving your efficiency by showing you the features of bash that are seldom discussed and often ignored.

Every day you spend many hours working in the shell, every little improvement in your worklflows will pay dividends many fold!

Learn more

What are IAM Roles for Kubernetes service accounts?

The IAM Roles for Kubernetes service accounts allow us to associate an IAM role with a Kubernetes service account. This feature is available through the Amazon EKS Pod Identity Webhook. The IAM role for a service account is then used to provide AWS credentials to the pod or resource using the service account. These credentials are automatically rotated before they expire.

If you have used WebIdentity federation with AWS Cognito, you will be familiar with the concept of IAM roles for service accounts.

Ok, so what do we need?

• OIDC Issuer - Already set up if you are running in EKS
• OIDC provider
• The IAM role that we want to associate with the service account
• The Kubernetes service account that we want to associate with the IAM role

Set up OIDC provider

The first step is to set up the OIDC provider. OIDC, if you are unfamiliar, is an authentication protocol similar to SAML, but it is based on JSON Web Tokens (JWT). AWS also uses it to authenticate users in AWS Cognito.

There are two parts to using OIDC an Issuer and a Provider. The issuer is the entity that issues the token, and the provider is the entity that validates the token. In our case, the issuer is handled by AWS, and Kubernetes handle the provider. When we create the EKS cluster, we already have an Issuer endpoint. If you want to use your OIDC issuer, you can provide that to EKS. But in this example, we’ll stick to the issuer already in the EKS cluster.

To get the URL of the issuer, run the following command:

1

aws eks describe-cluster --name temp-cluster --query "cluster.identity.oidc.issuer" --output text

You should see something like the following:

https://oidc.eks.us-east-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Let’s check first if an OIDC provider is already created for our cluster. Run the following command:

1

aws iam list-open-id-connect-providers

If you get an empty list, then you need to create the OIDC provider. Let’s define it in terraform:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

locals {
  oidc_issuer = "https://oidc.eks.us-east-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  # We'll use the cluster_namespace and the service_account_name later
  cluster_namespace = "my-namespace"
  service_account_name = "my-service-account"
}

# Set open IDC provider
data "tls_certificate" "cluster" {
  # If we created the EKS cluster in terraform, we would do something like the following
  # url = aws_eks_cluster.main.identity[0].oidc[0].issuer
  url = local.oidc_issuer
}

resource "aws_iam_openid_connect_provider" "cluster" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.cluster.certificates.0.sha1_fingerprint]
  # If we created the EKS cluster in terraform, we would do something like the following
  # url             = aws_eks_cluster.main.identity[0].oidc[0].issuer
  url = local.oidc_issuer
}

Now, if we run the command to list the OIDC providers, we should see the one we just created:

1

aws iam list-open-id-connect-providers

With the OIDC provider created, we can now create the IAM role and the Kubernetes service account.

Creating the IAM role

We will create a new IAM role, and we will call it eks-test-role. We want the pod to read a parameter from the SSM parameter store, so we will also create a policy for that and attach it to the same role.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

data "aws_region" "current" {}
data "aws_caller_identity" "current" {}

# SSM Parameter Store Service Role

resource "aws_iam_role" "pod_sa" {
  name = "eks-test-role"

  assume_role_policy = data.aws_iam_policy_document.pod_sa_assume_role_policy.json
}

data "aws_iam_policy_document" "pod_sa_assume_role_policy" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(local.oidc_issuer, "https://", "")}"]
    }
    condition {
      test     = "StringEquals"
      variable = "${replace(local.oidc_issuer, "https://", "")}:aud"

      values = ["sts.amazonaws.com"]
    }

    condition {
      test     = "StringEquals"
      variable = "${replace(local.oidc_issuer, "https://", "")}:sub"

      values = ["system:serviceaccount:${local.cluster_namespace}:${local.service_account_name}"]
    }
  }
}

We will create an additional policy allowing the pod to read any parameter in the parameter store. We are going to call it cluster-ssm-parameter-policy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

resource "aws_iam_role_policy_attachment" "sa_role_ssm_parameter_policy" {
  policy_arn = aws_iam_policy.ssm_parameter_policy.arn
  role       = aws_iam_role.pod_sa.name
}

resource "aws_iam_policy" "ssm_parameter_policy" {
  name = "cluster-ssm-parameter-policy"

  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Action" : [
          "ssm:DescribeParameters",
          "ssm:GetParameter",
          "ssm:GetParameters",
          "ssm:GetParametersByPath"
        ],
        "Effect" : "Allow",
        "Resource" : [
          "arn:aws:ssm:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:parameter/*"
        ]
      },
    ]
  })
}

We will create a go program that will serve as a basic HTTP server. The server’s sole function is to echo the parameter value. For Kubernetes to be able to pull the image from ECR, we need to create an ECR repository. Let’s do that. We are going to call it test-ecr:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

# ECR
resource "aws_ecr_repository" "main" {
  name = "test-ecr"
}

resource "aws_ssm_parameter" "ecr" {
  name  = "/ecr/ecr-url"
  type  = "String"
  value = aws_ecr_repository.main.repository_url
}

resource "aws_ecr_lifecycle_policy" "main" {
  repository = aws_ecr_repository.main.name

  policy = <<EOF
{
    "rules": [
        {
            "rulePriority": 1,
            "description": "Keep last 10 images",
            "selection": {
                "tagStatus": "tagged",
                "tagPrefixList": ["v"],
                "countType": "imageCountMoreThan",
                "countNumber": 10
            },
            "action": {
                "type": "expire"
            }
        }
    ]
}
EOF
}

The following is a quick and dirty go program that will read the parameter from the parameter store and echo it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

package main

import (
  "fmt"
  "log"
  "net/http"

  "github.com/aws/aws-sdk-go/aws"
  "github.com/aws/aws-sdk-go/aws/session"
  "github.com/aws/aws-sdk-go/service/ssm"
)

func main() {
  http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    // Create an SSM client
    sess, err := session.NewSession(&aws.Config{
      Region: aws.String("us-east-1"),
    })
    if err != nil {
      panic(err)
    }
    svc := ssm.New(sess)

    // Get the parameter
    param, err := svc.GetParameter(&ssm.GetParameterInput{
      Name: aws.String("/my/parameter"),
    })
    if err != nil {
      fmt.Fprintf(w, "Error:")
      fmt.Fprintf(w, err.Error())
    } else {
      // Print the value
      fmt.Fprintf(w, "Greeting: ")
      fmt.Fprintf(w, *param.Parameter.Value)
    }
  })

  fmt.Println("Starting server in port 8080")

  if err := http.ListenAndServe(":8080", nil); err != nil {
    log.Fatal(err)
  }
}

We will also need a Dockerfile to build the image:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

FROM golang:1.19-alpine

WORKDIR /usr/src/app

COPY go.mod go.sum ./

RUN go mod download && go mod verify

COPY main.go .
RUN go build -v -o /usr/local/bin/app ./...

EXPOSE 8080

CMD [ "app" ]

We can now build the image and push it to ECR. Let’s create a file called deploy.sh and add the following content:

1
2
3
4
5

#!/bin/bash
export AWS_ECR_URL=$(aws ecr describe-repositories | jq -r '.repositories[0].repositoryUri')
docker build -f Dockerfile . -t ${AWS_ECR_URL}:latest
aws ecr get-login-password | docker login --username AWS --password-stdin $(echo $AWS_ECR_URL | sed -r "s/\/.*//g")
docker push ${AWS_ECR_URL}:latest

We can now run the script:

1
2

chmod +x deploy.sh
./deploy.sh

And we should have our Image ready to deploy in Kubernetes.

Create the SSM parameter

We are going to create an SSM parameter that the pod will read. We are going to call it /my/parameter:

1
2
3
4
5

resource "aws_ssm_parameter" "my_parameter" {
  name  = "/my/parameter"
  type  = "String"
  value = "Hola Mundo"
}

Create the Kubernetes configuration

We will create the Kubernetes namespace.yml, deployment.yml, service.yml, and service-account.yml.

The namespace.yml file will create the namespace that we are going to use:

1
2
3
4

apiVersion: v1
kind: Namespace
metadata:
  name: my-namespace

The service-account.yml file will create the service account that we are going to use:

1
2
3
4
5
6
7

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
  namespace: my-namespace
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::<AWS_ACCOUNT_ID>:role/<ROLE_NAME>"

Replace <AWS_ACCOUNT_ID> and <ROLE_NAME> with the values you created in the previous steps.

The service.yml file will create the service that we are going to use:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

apiVersion: v1
kind: Service
metadata:
  name: sserver
  namespace: my-namespace
spec:
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  type: NodePort
  selector:
    app: sserver

The deployment.yml file will create the deployment that we are going to use:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sserver
  namespace: my-namespace
spec:
  selector:
    matchLabels:
      app: sserver
  replicas: 1
  template:
    metadata:
      labels:
        app: sserver
    spec:
      serviceAccountName: my-service-account
      containers:
        - name: sserver
          image: <ERC_URL>:latest
          ports:
            - containerPort: 8080

Replace <ERC_URL> with the value of the ECR we created in the previous steps.

Ok, now let’s deploy everything:

1
2
3
4

kubectl apply -f namespace.yml
kubectl apply -f service-account.yml  
kubectl apply -f service.yml
kubectl apply -f deployment.yml

And we should be ready to test.

Test that everything is working

We can now test if our pod uses the service account we assigned.

We can port forward to our local host and hit our endpoint to see if we get the parameter’s value. Assuming we only have one pod running in my namespace, we can do the following:

1

kubectl port-forward $(kubectl get pods -n my-namespace --no-headers | awk '{print $1}') 8080:8080 -n my-namespace

In a different terminal, we can do a curl to the endpoint:

1

curl localhost:8080

And we should see:

Greeting: Hola Mundo

Final thoughts

Setting up the OIDC might be the most confusing part if we haven’t worked with it before. But remember you need an Issuer, which is already provided if you use AWS EKS. You can link to an external issuer in the same way. We also need a provider accessible to our cluster, as we did.

After the OIDC part is completed, we only need to create our IAM roles as we usually do and build the association by creating a service account and assigning the role to it.

I hope this helps you to understand how to use IAM roles for service accounts in Kubernetes.

References

AWS Documentation - Kubernetes service accounts AWS Documentation - Well architected framework Illustrated guide to OAuth and OpenID Connect

In this post, I will assume that you already have an AWS account and have installed the aws cli, eksctl and kubectl on your machine.

This post will be short. It will only serve as a blueprint to get you started. I’ll be using the us-east-1 region, but you can use any region you want.

If you don’t know which region to use, you could try to use https://www.cloudping.info/ to check the latency between your machine and the AWS regions (It also works for other providers). There are many other factors that you should consider, like cost differences for resources in different regions, etcetera, but for this quick test, you could use latency.

SSH keys

We need ssh keys to access the cluster nodes. We will use the aws CLI to create a key pair and register it in the AWS Key Management Service (KMS).

aws ec2 create-key-pair --key-name temp-key --query 'KeyMaterial' --output text > temp-key.pem

Reference: AWS Documentation - create key pair

If you are unfamiliar with SSH keys, maybe read the first two sections of my article Understanding SSH Keys and using Keychain to manage passphrase on macOS.

The previous command will generate the private key and store it in temp-key.pem. To get the public key, you can run the following:

1

ssh-keygen -y -f temp-key.pem > temp-key.pub

Create the cluster

I prefer to keep my infrastructure as code. Even if we will be using eksctl in an imperative way, it is nice to reference back to where I have the initial settings. For that, we will be using a configuration file. Create a new file. I’ll name it cluster.yml with the following content:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: temp-cluster
  region: us-east-1

nodeGroups:
  - name: ng-1
    # the t3.medium supports 3 ENIs, so keep that in mind on how many pods you can put in the Node
    instanceType: t3.medium
    desiredCapacity: 2
    ssh:
      publicKeyPath: temp-key.pub

Now we can apply the configuration:

1

eksctl create cluster -f cluster.yml

The creation process takes a while, about 20 mins.

Connect to the cluster

Once the cluster is created, we want to set up kubectl to manage the cluster. We can do that by running the following:

1

aws eks --region us-east-1 update-kubeconfig --name temp-cluster

And finally, we can start working on the newly created cluster. Let’s check the services:

1

kubectl get svc

You should see something like the following:

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   10m

Perfect! We have a cluster up and running.

Now you can run your tests and destroy the cluster when you are done.

Modifying the Node Group

If you need to modify the Node Group, you can do that by editing the cluster.yml file and running:

1

eksctl update cluster -f cluster.yml

Remember that not all aspects of the node group can be modified, for example, the instance type. If you need to change the instance type, you must create a new node group and delete the old one.

Destroy the cluster

To clean up the cluster, we can run the following:

1

eksctl delete cluster -f cluster.yml

And that’s it! Happy hacking!

Final thoughts

This was a quick and simple post to learn how to create your Kubernetes cluster and start learning. The instructions here are straightforward. The interesting part is when you start modifying your cluster and node group configuration.

Anyways, I hope it was helpful.

References

• https://www.cloudping.info/ to check the latency between your machine and the AWS regions.
• AWS Documentation - create key pair
• Understanding SSH Keys and using Keychain to manage passphrase on macOS
• eksctl documentation - the schema with all the options for your cluster.yml
• AWS Documentation - deleting an Amazon EKS cluster

I’ll describe how to create a Kubernetes cluster in Amazon EKS using Terraform in this article.

Note: If you are interested in learning more about how to set up the directory structure for your Terraform project, you might find my guide, Meditations on Directory Structure for Terraform Projects, useful.

Create the EKS cluster

The eks_cluster resource is the one that creates the EKS cluster. It is a simple resource. Its required fields are name, role_arn and vpc_config.

• name is the name of the cluster
• role_arn is the ARN of the IAM role that the cluster will use
• vpc_config includes the VPC ID and the subnets that the cluster will use

With that information, we should be able to create the cluster. In reality, if we set up the eks_cluster resource, we’ll have a Kubernetes cluster, but we still need to set up the infrastructure for the worker nodes. EKS manages the control plane, so once we’ve created the EKS cluster, AWS will handle the control plane for us, so we don’t have to worry about that. But we are still responsible for setting up the worker nodes. We’ll do that later. Let’s start by exploring what each of the fields for the eks_cluster requires (except name because naming is the hardest part of programming and is out of this article’s scope).

IAM role

A Kubernetes cluster needs to run on Nodes. Those nodes are EC2 instances, and they need to have permission to run the Kubernetes components. Those permissions are defined in an IAM role, and that role is attached to the EC2 instances that run the Kubernetes components.

The IAM role should allow the cluster to create those nodes (EC2 instances), create load balancers (Ingresses in k8s), manage auto scaling groups, etcetera. So we need to grant permissions to the EKS cluster to create, modify and update all the resources it needs. The good thing is that Amazon already made a policy we can use that handles the basics, arn:aws:iam::aws:policy/AmazonEKSClusterPolicy. You can see how the policy is defined if you log into your AWS console and search for it in the IAM section or by using the following command using the aws cli:

1

aws iam get-policy --policy-arn arn:aws:iam::aws:policy/AmazonEKSClusterPolicy

From there, you’ll be able to see the DefaultVersionId. To view the default version of the policy, you can use the following command:

1

aws iam get-policy-version --policy-arn arn:aws:iam::aws:policy/AmazonEKSClusterPolicy --version-id v5

Ok, so let’s create the IAM role and attach the policy. We can do that using the aws_iam_role and aws_iam_role_policy_attachment resources.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# EKS Cluster IAM Role
resource "aws_iam_role" "cluster" {
  name = "eks-cluster-role"

  assume_role_policy = data.aws_iam_policy_document.eks_assume_role_policy.json
}

data "aws_iam_policy_document" "eks_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["eks.amazonaws.com"]
    }
  }
}

resource "aws_iam_role_policy_attachment" "cluster_AmazonEKSClusterPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.cluster.name
}

With the IAM role out of the way, let’s now look at the vpc_config.

VPC configuration and subnets

The VPC configuration has more moving parts. There are specific requirements that the VPC needs to meet for the cluster to work properly. The requirements are specified in the AWS Documentation - Amazon EKS VPC and subnet requirements and considerations. I won’t rewrite what is already explained in the documentation, but there are a few items worth noting:

• The VPC must have at least two subnets that are in different availability zones
• The VPC must have DNS hostname and DNS resolution support.

Those are the main ones. Other considerations include having enough IP addresses, etcetera, so check the documentation.

Let’s define the VPC and the subnets. We’ll use the aws_vpc and aws_subnet resources.

I won’t go into detail about how to plan the CIDR blocks and how to do your subnets, but I’ll show you the schema I’ll use:

VPC CIDR:
10.0.0.0/18
  public_subnets = {
    a = "10.0.0.0/22"
    b = "10.0.4.0/22"
  }

  private_subnets = {
    a = "10.0.8.0/22"
    b = "10.0.12.0/22"
  }

If you are looking for a subnet calculator, you can use mine: https://rdicidr.rderik.com/. The observant reader will notice that I like palindromes, and RDICIDR is one! it stands for RDerik Interactive CIDR.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113

data "aws_region" "current" {}

resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/18"

  # EKS requirements
  # The VPC must have DNS hostname and DNS resolution support. Otherwise, nodes can't register to your cluster.
  # https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html
  enable_dns_hostnames = true
  enable_dns_support   = true
}

resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.main.id
}

resource "aws_default_route_table" "public" {
  default_route_table_id = aws_vpc.main.default_route_table_id


  route = [
    {
      cidr_block = "0.0.0.0/0"
      gateway_id = aws_internet_gateway.main.id

      # these seem to be required due to an AWS provider bug
      carrier_gateway_id         = ""
      destination_prefix_list_id = ""
      egress_only_gateway_id     = ""
      instance_id                = ""
      ipv6_cidr_block            = ""
      local_gateway_id           = ""
      nat_gateway_id             = ""
      network_interface_id       = ""
      transit_gateway_id         = ""
      vpc_endpoint_id            = ""
      vpc_endpoint_id            = ""
      vpc_peering_connection_id  = ""
    }
  ]
}

resource "aws_subnet" "public" {
  for_each = {
    a = "10.0.0.0/22"
    b = "10.0.4.0/22"
  }

  vpc_id                  = aws_vpc.main.id
  availability_zone       = "${data.aws_region.current.name}${each.key}"
  cidr_block              = each.value
  map_public_ip_on_launch = true
}

resource "aws_route_table_association" "public" {
  for_each       = aws_subnet.public
  subnet_id      = each.value.id
  route_table_id = aws_vpc.main.default_route_table_id
}

resource "aws_eip" "main" {
  for_each = aws_subnet.public
  vpc      = true
}

resource "aws_nat_gateway" "main" {
  for_each      = aws_subnet.public
  allocation_id = aws_eip.main[each.key].id
  subnet_id     = each.value.id
}


resource "aws_subnet" "private" {
  for_each = {
    a = "10.0.8.0/22"
    b = "10.0.12.0/22"
  }

  vpc_id            = aws_vpc.main.id
  availability_zone = "${data.aws_region.current.name}${each.key}"
  cidr_block        = each.value
}

resource "aws_route_table" "private" {
  for_each = aws_nat_gateway.main
  vpc_id   = aws_vpc.main.id

  route = [
    {
      cidr_block     = "0.0.0.0/0"
      nat_gateway_id = each.value.id

      # these seem to be required due to an AWS provider bug
      carrier_gateway_id         = ""
      destination_prefix_list_id = ""
      egress_only_gateway_id     = ""
      gateway_id                 = ""
      instance_id                = ""
      ipv6_cidr_block            = ""
      local_gateway_id           = ""
      network_interface_id       = ""
      transit_gateway_id         = ""
      vpc_endpoint_id            = ""
      vpc_peering_connection_id  = ""
    }
  ]
}

resource "aws_route_table_association" "private" {
  for_each       = aws_subnet.private
  subnet_id      = each.value.id
  route_table_id = aws_route_table.private[each.key].id
}

We now have a VPC with two public and two private subnets. The public subnets have a route to the internet through an internet gateway, and the private subnets have a route to the internet through a NAT gateway.

EKS Cluster security group

For the VPC configuration, we will need to provide the security groups for the EKS cluster. We’ll use the aws_security_group resource. You will notice that in the following security group definition, we are referencing aws_securitygroup.eks_nodes.id, which hasn’t been created yet. We’ll create it later when we define the nodes. At the moment, assume that it’ll be defined later.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# EKS Cluster Security Group
resource "aws_security_group" "eks_cluster" {
  name        = "eks-cluster-sg"
  description = "Cluster communication with worker nodes"
  vpc_id      = aws_vpc.main.id
}

resource "aws_security_group_rule" "cluster_inbound" {
  description              = "Allow worker nodes to communicate with the cluster API Server"
  from_port                = 443
  protocol                 = "tcp"
  security_group_id        = aws_security_group.eks_cluster.id
  source_security_group_id = aws_security_group.eks_nodes.id
  to_port                  = 443
  type                     = "ingress"
}

resource "aws_security_group_rule" "cluster_outbound" {
  description              = "Allow cluster API Server to communicate with the worker nodes"
  from_port                = 1024
  protocol                 = "tcp"
  security_group_id        = aws_security_group.eks_cluster.id
  source_security_group_id = aws_security_group.eks_nodes.id
  to_port                  = 65535
  type                     = "egress"
}

Now let’s look at creating the nodes where the Kubernetes cluster will run.

Creating the worker nodes

We have a few different ways to set up the worker nodes. We could use two approaches:

• Self-managed nodes - We would need to manage the nodes ourselves using Auto Scaling Groups and all that it entails.
• Managed node groups - “Amazon EKS managed node groups automate the provisioning and lifecycle management of nodes (Amazon EC2 instances) for Amazon EKS Kubernetes clusters.” Amazon EKS nodes

We’ll use Managed node groups. The reason is that it’s easier to manage and the recommended way to do it.

Let’s start by creating the IAM role that the nodes will use. We’ll call it eks-node-role and attach the AmazonEKSWorkerNodePolicy and AmazonEKS_CNI_Policy policies to it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# EKS Node IAM Role
resource "aws_iam_role" "node" {
  name = "eks-node-role"

  assume_role_policy = data.aws_iam_policy_document.ec2_assume_role_policy.json
}

data "aws_iam_policy_document" "ec2_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

Now we’ll attach the policies to the role.

1
2
3
4
5
6
7
8
9

resource "aws_iam_role_policy_attachment" "node_AmazonEKSWorkerNodePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.node.name
}

resource "aws_iam_role_policy_attachment" "node_AmazonEKS_CNI_Policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.node.name
}

If you plan to use the AWS Load Balancer controller, and you want to directly assign the policy to the nodes, we can do that now:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

resource "aws_iam_policy" "alb_controller_policy" {
  name = "AlbControllerPolicy"
  # We are going to use the ALB controller implementation from the Kubernetes SIGs
  # the following policy is needed
  # Source: `curl -o alb_controller_policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.4.4/docs/install/iam_policy.json`
  policy = file("${path.module}/alb_controller_policy.json")
}

resource "aws_iam_role_policy_attachment" "node_alb_controller_policy" {
  policy_arn = aws_iam_policy.alb_controller_policy.arn
  role       = aws_iam_role.node.name
}

Now we’ll create the security group for the nodes. We’ll call it eks-node-sg, and we’ll allow traffic from the security group of the control plane.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

# EKS Node Security Group
resource "aws_security_group" "eks_nodes" {
  name        = "eks-node-sg"
  description = "Security group for all nodes in the cluster"
  vpc_id      = aws_vpc.main.id

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_security_group_rule" "nodes_internal" {
  description              = "Allow nodes to communicate with each other"
  from_port                = 0
  protocol                 = "-1"
  security_group_id        = aws_security_group.eks_nodes.id
  source_security_group_id = aws_security_group.eks_nodes.id
  to_port                  = 65535
  type                     = "ingress"
}

resource "aws_security_group_rule" "nodes_cluster_inbound" {
  description              = "Allow worker Kubelets and pods to receive communication from the cluster control plane"
  from_port                = 1025
  protocol                 = "tcp"
  security_group_id        = aws_security_group.eks_nodes.id
  source_security_group_id = aws_security_group.eks_cluster.id
  to_port                  = 65535
  type                     = "ingress"
}

resource "aws_security_group_rule" "nodes_cluster_outbound" {
  description              = "Allow worker Kubelets and pods to receive communication from the cluster control plane"
  from_port                = 1025
  protocol                 = "tcp"
  security_group_id        = aws_security_group.eks_nodes.id
  source_security_group_id = aws_security_group.eks_cluster.id
  to_port                  = 65535
  type                     = "egress"
}

And we finally have everything to create our EKS cluster.

Creating the EKS cluster:

We’ll create the cluster using the eks_cluster resource. We’ll call it eks-cluster and use everything we created before. But we also want to log events in CloudWatch for the cluster, so let’s create the log group first:

1
2
3
4
5

resource "aws_cloudwatch_log_group" "cluster" {
  # Reference: https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html
  name              = "/eks/eks-cluster/cluster"
  retention_in_days = 0
}

Ok, now we are ready to put everything together and set up the EKS cluster:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

# EKS Cluster
resource "aws_eks_cluster" "main" {
  name = "eks-cluster"

  # reference for log_types:
  # https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html
  enabled_cluster_log_types = ["api", "audit"]

  role_arn = aws_iam_role.cluster.arn

  # reference for EKS versions:
  # https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html
  version = "1.23"

  vpc_config {
    # Security Groups considerations reference:
    # https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html
    security_group_ids      = [aws_security_group.eks_cluster.id, aws_security_group.eks_nodes.id]
    subnet_ids              = concat(aws_subnet.public.*.id, aws_subnet.private.*.id)
    endpoint_private_access = true
    endpoint_public_access  = true
    public_access_cidrs     = ["0.0.0.0/0"]
  }

  depends_on = [
    aws_iam_role_policy_attachment.cluster_AmazonEKSClusterPolicy,
    aws_cloudwatch_log_group.cluster
  ]
}

Now we can happily terraform apply, and we’ll have our EKS cluster up and running.

Configuriong kubectl to connect to the cluster

Now that we have our cluster up and running, we need to configure kubectl to connect to it. We can do that by running the following command:

1

aws eks --region us-east-1 update-kubeconfig --name eks-cluster

Modify the region and cluster name if you choose different ones.

Final thoughts

Using Terraform to create the EKS cluster seems more convoluted than just typing a couple of lines with eksctl, but it has the benefit of describing what was deployed and having it as part of our Infrastructure as Code.

I hope this article was helpful and shed some light on how to set up your initial EKS cluster.

References

• AWS Documentation - Amazon EKS VPC and subnet requirements and considerations
• AWS Load Balancer controller
• AWS documentation - EKS create cluster
• Amazon - EKS nodes
• CIDR Calculator - https://rdicidr.rderik.com/
• Terraform resoruce -aws_vpc
• Terraform resource - aws_iam_role and aws_iam_role_policy_attachment
• Terraform resource - aws_security_group
• Terraform resource - eks_cluster
• Terraform resource - eks_cluster
• Terraform resource -aws_subnet