I've been hearing a lot about this new thing recently GitOps, then I realized like most trends oh I've been doing this for the past several years.

The basic gist of this is that infrastructure is defined and maintained through git, and a related pipeline. That is to say a merge request is used to modify infrastructure then deploy.

I recently started digging into EKS and GitLab, and saw there was a bit of initial setup. So wanted to run vet how to create this.

Branching Strategy

Any good git based strategy is only as strong, or weak, as it's' branching strategy. Setting up a strong boundary of how branches, and merges occur is tantamount to success.

  • master The stable release of the infrastructure
  • region/us-east-2, region\/\w+-\w+-\d+ The region to deploy to
  • dev/remove_password_protection_and_chmod_777_all_the_things The development feature to be merged in added.

Master

By stable we mean you have vetted out adding a new user, vpc or some other feature. It is then ready to be marked as stable. This will version you're infrastructure.

Version 1

Add CI/CD IAM Role

Version 1.1

Create RDS instances, one per environment staging, dev, prod, etc.
Provision Secondary IAM role for RDS instances, isolated to each environment table.

I haven't determined what is a breaking change, per semantic version. But The idea is providing a change log of what a release contains. This also provides the ability to trace when infrastructure was changed.

Master will always be the latest stable version. While a tag will be created for prior versions

Region

When merging master or a stable tag into this branch, the release at that version will be applied to the target region. By having a branch name of region/us-east-2 or region/us-west-2, you're being explicit of what goes into a region. Combined with pipelines you can trace what was deployed into that target region.

Merge Flow

graph TD; f1[NEW_FEATURE] --> m[master] m[master] --> r[region/us-east-2]

There are two flows you can take with this. Either every feature must be merged into master. Then master into a region, after having been approved. Or you merge the feature into master and region.

graph TD; f1[NEW_FEATURE] --> m[master] f1[NEW_FEATURE] --> r[region/us-east-2]

CI/CD

I will be using GitLab, as usual with this article. In first testing this I found you couldn't set  a variable dynamically in before_script.

#!/usr/bin/env bash

region_pattern=region\/\w+-\w+-\d+
AWS_DEFAULT_REGION=''
if $(echo $CI_COMMIT_REF_NAME |grep -qE 'region/\w+-\w+-[0-9]'); then AWS_DEFAULT_REGION=`echo $CI_COMMIT_REF_NAME|sed s'/region\///g'`; else echo "Region not found"; fi
if [ $AWS_DEFAULT_REGION == '' ]; then echo "Region not found exiting."; exit 1; fi
echo "$AWS_DEFAULT_REGION"

In context it should be used as follows. This will pull the default region out of the branch name. Additionally this CI section will only run on branches matching the AWS region regex.

:bootstrap:iam_roles:
  stage: configuration
  when: manual
  only:
    - /^region/\w+-\w+-[0-9]$/
  script:
    - cdk deploy

Once we have the region, we can provide the AWS access key and secret as masked environment variables.

Intro to CDK

I'm going to go more into this in part 2. But suffice it to say CDK is a programmatic way to generate cloud formation templates.

The end goal, is to boot strap an AWS region and account for GitLab EKS integration. The requirements of which are several IAM roles, and a vpc.

suspend fun main(args: Array<String>) {
    println("Received a default region of: ${EnvConfig.awsDefaultRegion}")
    val awsApp = App()
    val environment= EnvironmentBuilder()
        .account(EnvConfig.rootAccount)
        .region(EnvConfig.awsDefaultRegion)
        .build();
    println("Target environment is:")
    println(environment)
    Stack(awsApp, "EKSBootStrap", StackProps.builder()
        .env(environment).build())
    awsApp.synth()
}
class Stack(
    private val parent: Construct?,
    private val id: String?,
    private val props: StackProps? = null
) : Stack(parent, id, props) {
    init {
        // Users
        val gitLabCICDUser = User.Builder.create(this, "gitlab-cicd")
            .userName("gitlab-cicd")
            .build()
    }
}

Here we define a basic stack, and build an environment based on the default region noted above. The target account is also coming through as an environment variable.

Once we have defined the cloud formation stack, we build the actual stack. While you can specify multiple stacks, I have been breaking out each stack into a seperate gradle module.

The notation is a typical java builder notation. The CDK docs are overall very helpful, and easy to follow along with. Of note is we don't have to do anything with the returned user. It is now added to the stack. Passing in this adds it into the stack notation. The user can then be passed as an argument later in the stack allowing for referencing multiple parts of the configured stack.

If you wish to seperate out items into seperate files, i.e. user, roles, policies each in different files. That can be done as shown below. The stack is equivalent to this.

const val gitLabEKSRoleName = "GitLabEKSRole"
fun createGitLabEKSRole(stack: Construct): Role =
    Role.Builder.create(stack, gitLabEKSRoleName)
        .roleName(gitLabEKSRoleName)
        .build()

Gradle Differences

One of the big differences in using kotlin, is the java has a default archetype for maven. We just need to wire up cdk and gradle to work together.

application Plugin

application {
    mainClassName = "design.animus.aws.eks.bootstrap.DirectorKt"
}

After defining the plugin to run as an application. We now need to configure cdk. Assuming this file is stored at $repoRoot/eks_bootstrap, where $repoRoot is the root of the repository.

{
  "app": "../gradlew :eks_boostrap:run",
  "context": {
    "@aws-cdk/core:enableStackNameDuplicates": "true",
    "aws-cdk:enableDiffNoFail": "true"
  }
}

The relative path tells it how to get back to the gradle wrapper. So dependent on how you adjust your project/module structure adjust the pathing as necessary.

cd eks_bootstrap
cdk diff
cdk deploy
# Or
cdk deploy --require-approval=never 

Once you have changed into the target directory, you can now run cdk. This will execute the gradle application. Building out the cloud formation, the target directory for these generated templates is cdk.out.

  • cdk diff Prints the differences, or what will change if the cloud formation template is applied.
  • cdk deploy Will deeploy the template, but list the changes first, then prompt you to approve the changes.
  • cdk deploy --require-approval=never Is for CI implementations, as it will not prompt you for approval.

Next Up

Now that we've laid the ground work, in the next piece I'll go over what EKS requires. Then get to creating a cluster and binding to GitLab.