Kubernetes 0.0.1

Summary

Type safe kotlin records for Kubernetes

  • Multi platform supports JVM, JS, Linux
  • Generated off the latest schemas, pulled from the Kubernetes repository
  • Support for 1.16.x, 1.17.x, 1.18.x, 1.19.x, 1.20.x

Why

Now

I had been using Fabric8 and Kotlin K8 DSL, to build out  manifests for Kubernetes. However I found a few pain points.

  • Hard to interopt with custom resource definitions.
  • Lack of testing against various versions
  • Long time to execute deployments.
  • Hard interopt with non JVM teams.
  • Long build / run times for deployment. Fabric 8 was taking around five minutes for me, while my current CDK8s deploy is around 30 seconds.

When falling back to address these items it was usually solved with an open api generator. That was lacking on the JVM side, and didn't create idiomatic Kotlin. Which was the reason for developing the JSON Schema and Open API modules. Those two have acted as the back bone for this module.

Compared to CDK8s, there is less over arching differences. The biggest difference would be the multi platform support. Hypothetically if you work in a company with JVM and JS. You can define templates in common and export to both languages.

Working on these manifests it also feels like a recompilation isn't needed every time. As I work on deployments after the initial determining of resources, configuration. The only thing that really changes each pipeline/deployment run is the image tag. There is no need at that point to re-compile the deployment script.

Longer Term

With the core modules provided from JSON schema. It allows for easy diffing between deployments. Showing what if any changes has been made. With the strong typing per version if you upgrade your cluster from 1.18.x -> 1.20.x. Simply increment the dependencies, and the compiler will tell you where any changes are needed.

I also wanted to focus on more type safety of not just deployment manifests, but also CRD and other cloud infrastructure. I forsee this also tying into a potential operator to track release quality and observability.

Modules

  • kubernetes-common
  • kubernetes-11X
  • kubernetes-11X-mapper
  • kubernetes-crd-generator

There are modules for each respective released kubernetes version, and the types are compiled based on the latest tag from the upstream repository.

The versions are flattend to 120 and not 1.20.7. As this is a flat versioned mono repo, in the future updates to schemas will result in bumping of the core artifact. I.E. kubernetes-120:0.0.1 -> kubernetes-120:0.0.2. The change log notes will denote the targeted kubernetes schemas.

kubernetes-common

Holds common records, namely

@Serializable
data class UnknownKubernetesRecord(val apiVersion: String, val kind: String)

Which allows determining what kubernetes resources to generate. There are also helper methods for handling yaml to json translation. This is used to interopt with existing resources.

kubernetes-11X

Provides the core types for the kubernetes API, examples below.

kubernetes-11X-mapper

Is used to map an unknown record to an actual record type. Taking in the above UnknownKubernetesRecord and providing a known serializer.

@ExperimentalSerializationApi
public fun determineSerializerForKubernetesRecord(record: UnknownKubernetesRecord):
    AJSONSchemaSerializer<IJSONSchemaObjectBase> {
  val serializer = when (record) {
    UnknownKubernetesRecord(apiVersion="admissionregistration.k8s.io/v1",
        kind="MutatingWebhookConfiguration" ) -> MutatingWebhookConfigurationSerializer
        ...
 }

This is most useful when you  want to parse a directory of yaml files, or may not known the resource you're attempting to parse.

val sampleResource = File("./some-k8-resource.json").readText()
val unknownRecord = parseUnknownKubernetesRecord(sampleResource)
val serailizer = determineSerializerForKubernetesRecord(unknownRecord)

kubernetes-crd-generator

Is a preview feature which will generate Kotlin code from CRDs, see below.

Current Use Case

@ExperimentalSerializationApi
@Serializable
data class Stack(
    val kubernetesResources: List<@Serializable(with= IKubernetesSerializer::class) IKubernetes>
) : IJSONSchemaObjectBase 

@ExperimentalSerializationApi
suspend fun getStack(): Stack {
    return Stack(
        kubernetesResources = listOf(
            testServiceService,
            createTestServiceDeployment("services-test-api")
        )
    )
}

Define a stack, containing a list of resources. At this time only Kubernetes is supported, but I'm working on other cloud providers.

Each of these resources has a method getAJSONSchemaSerializer which will tell us how to serialize the resource. The main method would look as follows.

suspend fun main() {
  val stack = getStack()
  stack.kubernetesResources.forEachIndexed { index, kubeResource ->
    File("./dist-stack/kubernetes/$index-kubernetes.json").writeText(
      Json.encodeToString(kubeResource.getAJSONSchemaSerializer(), kubeResource)
    )
  }
}

Unfortunately for the time being a manual kubectl apply is necessary.

kubectl apply -f ./dist-stack/kubernetes/*

With the JSON Diff you can also do a difference to see what's changed between deployments. The last deployed stack will need to be saved in a queryable store.

    val oldStack = getLastDeployedStack()
    val newStack = getStack()
    val changes = diffJsonObjects(oldStack, newStack) ?: listOf()
    changes.forEach {
        println(it)
    }

Defining Resources

There are no DSL builders at this time, instead a builder pattern is supported.

@ExperimentalSerializationApi
suspend fun createTestServiceDeployment(name: String): Deployment {
    val commonAnnotations = ObjectMetaAnnotations(
        gitLabDeploymentAnnotations + gitLabRepoAnnotations
    )
    return Deployment(
        apiVersion = "apps/v1",
        kind = "Deployment",
        metadata = ObjectMeta(
            annotations = commonAnnotations,
            name = "$namePrefix-deployment",
            namespace = GitLabKubeConfig.kubeNamespace
        ),
        spec = DeploymentSpec(
            selector = LabelSelector(
                matchLabels = LabelSelectorMatchLabels(selectorMap)
            ),
            template = PodTemplateSpec(
                metadata = ObjectMeta(
                    annotations=commonAnnotations,
                    labels = ObjectMetaLabels(selectorMap)
                ),
                spec = PodSpec(
                    containers = listOf(
                        createContainerFromGitLab(name).copy(
                            ports = listOf(ContainerPort(containerPort = 8080.toShort()))
                        )
                    ),
                    imagePullSecrets = listOf(
                        LocalObjectReference(name = "gitlab-registry")
                    )
                )
            )
        )
    )
}
@ExperimentalSerializationApi
val testServiceService = Service(
    apiVersion="v1",
    kind="Service",
    metadata = ObjectMeta(
        name = "$namePrefix-svc",
        namespace = GitLabKubeConfig.kubeNamespace
    ),
    spec = ServiceSpec(
        selector = ServiceSpecSelector(
            selectorMap
        ),
        ports = listOf(
            ServicePort(
                protocol = "TCP",
                port = appServicePort,
                targetPort = appServicePort.toInt()
            )
        )
    )
)

CRD Generator

As operators become more and more common in Kubernetes. There is a rise in custom resources. A gradle plugin is offered to build the custom resources into kotlin types. This example is based on the PodMonitor from kube prometheus stack.

  • This will require the resources to be stored or retrieved ahead of time.
plugins {
    id ("design.animus.kotlin.mp.schemas.kubernetes.crd.generator") version "0.0.1-SNAPSHOT"
}

CRDGeneratorConfig {
    crdConfig = design.animus.kotlin.mp.schemas.kubernetes.crd.CRDConfig(
        packageBase = "design.animus.kotlin.mp.schemas.kubernetes.test.crd",
        outputPath = File("$projectDir/generated/"),
        crds = listOf(
            CRD(
                path=File("$projectDir/main/resources/podmonitors.monitoring.coreos.com.json"),
                name="PodMonitor"
            )
        )
    )
}
 ./gradlew :test:crd:CRDGenerate

That will then provide a PodMonitor following the other standards.

@Serializable
@ExperimentalSerializationApi
public data class PodMonitor(
  public val apiVersion: String? = null,
  public val kind: String? = null,
  public val metadata: PodMonitorMetadata? = null,
  public val spec: @Serializable(with = PodMonitorSpecSerializer::class) PodMonitorSpec
) : IPodMonitor, IJSONSchemaObjectBase {
  public override fun getAJSONSchemaSerializer(): AJSONSchemaSerializer<IJSONSchemaObjectBase> =
      PodMonitorSerializer as AJSONSchemaSerializer<IJSONSchemaObjectBase>
}

Extensions

GitLab

There are some common repeated practices in deploying Kubernetes logic. I've re-written the same code at several firms as  I've provided services. As GitLab was my more common deployment pipeline I had re written the below several times.

Extensions follow the standards of the other projects. They are seperate modules you may choose to include.

All common environment variables are provided as Objects, typed as best as possible.

object GitLabEnvConfig {
  val ciProjectPathSlug: String by Environment("CI_PROJECT_PATH_SLUG")
  val ciEnvironmentSlug: String by Environment("CI_ENVIRONMENT_SLUG")
  val ciJobId: String by Environment("CI_JOB_ID")
  val ciCommitSha: String by Environment("CI_COMMIT_SHA")
  val ciCommitRefSlug: String by Environment("CI_COMMIT_REF_SLUG")
  val ciCommitShortSha: String by Environment("CI_COMMIT_SHORT_SHA")
  ... 
}

GitLabEnvConfig provides all CI/CD variables.  While GitLabKubeConfig pulls variables set via the kubernetes deploy boards.

object GitLabKubeConfig {
  val kubeUrl: String by Environment("KUBE_URL")
  val kubeToken: String by Environment("KUBE_TOKEN")
  val kubeNamespace: String by Environment("KUBE_NAMESPACE")
  val kubeCaPemFile: String by Environment("KUBE_CA_PEM_FILE")
  val kubeConfig: String by Environment("KUBECONFIG")
  val kubeIngressBaseDomain: String by Environment("KUBE_INGRESS_BASE_DOMAIN")
}

A create container method is provided which follows similar notation to their documentation. Note if you host a private image registry a im

@ExperimentalSerializationApi
/**
 * Assumes the following '$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG/$name:$CI_COMMIT_SHA'
 */
suspend fun createContainerFromGitLab(name: String): Container {
  return Container(
    name = name,
    image = "${GitLabEnvConfig.ciRegistryImage}/${GitLabEnvConfig.ciCommitRefSlug}/$name:${GitLabEnvConfig.ciCommitSha}"
  )
}
val gitLabDeploymentAnnotations = mutableMapOf<String, String>(
  "app.gitlab.com/app" to GitLabEnvConfig.ciProjectPathSlug,
  "app.gitlab.com/env" to GitLabEnvConfig.ciEnvironmentSlug,
)

val gitLabRepoAnnotations = mutableMapOf(
  "gitlab/projectName" to GitLabEnvConfig.ciProjectName,
  "gitlab/commitSha" to GitLabEnvConfig.ciCommitSha,
  "gitlab/projectUrl" to GitLabEnvConfig.ciProjectUrl
)

Lastly there are the annotations that are common to enable the deploy boards. These annotations were used in the above deployment example.