Prior to going into a deep dive into JSON schema support, and how it works. I thought it would be beneficial to go over what value does this provide. Looking back at my post language of the cloud, highlights several of the benefits of JSON Schema.

Goals

  • Parse the open API specification into a type safe model.
  • Use that type safe model to generate out models for an Open API specification.
  • Parse the Kubernetes Open API specification to generate out kubernetes  models.

Caveats

  • This is running via a Gradle plugin. I realize gradle is bad and am working on a self executable CLI.
  • Most code has not been merged to master. It is primarily under a branch dev/initial. This is a preview post

Pre-Requisites

pluginManagement {
    resolutionStrategy {
        eachPlugin {
            if (requested.id.id.startsWith("design.animus.kotlin.mp.contract.json_schema_generator")
            ) {
              useVersion("0.0.1-SNAPSHOT")
            }
        }
    }
    repositories {
        mavenLocal()
        mavenCentral()
        jcenter()
        gradlePluginPortal()
        maven { url = java.net.URI("https://dl.bintray.com/kotlin/kotlinx") }
        maven { url = java.net.URI("https://kotlin.bintray.com/kotlin-dev") }
        maven { url = uri("https://animusdesign-repository.appspot.com") }
    }
}

Firstly you will need to enable my repository for the gradle plugin. Then set the version to match the published artifact, which at this time is 0.0.1-SNAPSHOT, open api follows a similar version.

JSON Schema Plugin

  id("design.animus.kotlin.mp.contract.json_schema_generator")

Open API Plugin

  id("design.animus.kotlin.mp.schemas.open_api_generator")

Open API

The open api specification is available via Github. There are several options available for this. As an initial quick take I am storing the schema in a the repository locally.

Next we configure the plugin.

JsonSchemaGeneratorConfig {
  schema = listOf(
    GeneratorConfig(
      packageBase = "design.animus.kotlin.mp.schemas.openapi.generated",
      outputPath = File("$projectDir/generated/commonMain/kotlin"),
      schemaFile = File("$projectDir/src/commonMain/resources/openapi-3.0.schema.json"),
      schemaName = "OpenAPI"
    )
  )
}

If you've been following kotlin-frm, this follows a very similar pattern.

  • packageBase Set's the base package name for the generated artifacts.
  • outputPath Set's where the generated code will live. The generated code is common and platform agnostic.
  • schemaFile Is the path to the schema.
  • schemaName Set's the class name that will act as the root of the schema. I will cover more in how json schema works in the deep dive.

Note

If you are outputting the generated code outside of  src/main/kotlin or another standard location. That folder needs to be added to the target source set. i.e. From my convention I store generated code under generated and add it too git ignore.

kotlin {
  ...
  sourceSets {
    val commonMain by getting {
      kotlin.setSrcDirs(mutableListOf("src/commonMain/kotlin", "generated/commonMain/kotlin"))
      }
      ...
  }
  ...
}

Running the Generator

./gradlew :open_api:jsonSchemaGenerate

You can run the plugin either via gradlew, or the drop down menu in Intellij.

Calling the Serializer

    // Load the Open API Json Document
    val raw = loadTestSchema("petstore.api.json")
    // Parse the Document into a kotlin object
    val schema = Json.decodeFromString(OpenAPI.serializer(), raw)

JSON schema is complex, and there are a number of incongruities with the kotlin type system. Due to this a common JSON Schema serializer has been created, which will automatically create serializer's for nested objects. Summing this up in two points.

  • The root objects may be serialized via the typical $className.serializer()
  • Any nested objects, outside of the root context will be serialized with $classNameSerializer

A root object is defined as the main object in the schema. That is to say the schema file it self is a definition for an object. In this case the object is OpenAPI. Any nested types. Say path definition, security, etc. Are different and require a generated serializer object to built.

There is a test in the [open-api repo](https://gitlab.com/AnimusDesign/schemas/open_api/). Which demonstrates how to parse and validate schemas.

Digesting Kubernetes

The upstream spec is locate here.

As an important note this is in Swagger 2.0, and will need to be converted to OpenAPI 3.0. I have concvereted and stored version 1.19 in my repo for test purposes.

See this stack over flow question. Which basically amounts too. The difference is in openapi instead of openapi-yaml. This will provide json insteasd of yaml.

java -jar swagger-codegen-cli-3.0.19.jar generate
     -l openapi
     -i https://petstore.swagger.io/v2/swagger.yaml
     -o OUT_DIR

Open API Plugin

An open api plugin was generated which matches very closely to the JSON Schema api plugin.

plugins {
  id("design.animus.kotlin.mp.schemas.open_api_generator")
}

OpenAPIGeneratorConfig {
  schemas = listOf(
    GeneratorConfig(
      packageBase = "design.animus.kotlin.mp.schemas.kubernetes.generated",
      outputPath = File("$projectDir/generated/commonMain/kotlin"),
      schemaFile = File("$projectDir/src/commonMain/resources/kubernetes.119.openapi.json"),
      schemaName = "Kubernetes",
      createBaseObject = false,
      mutateObjectName = Option.Some { renameContext ->
        // Expected name like Follows
        // io.k8s.kube-aggregator.pkg.apis.apiregistration.v1beta1.ServiceReference
        // We want just the last piece
        renameContext.itemName.split(".").last()
      },
      mutatePackageName = Option.Some { renameContext ->
        // see package base above for starting point
        // Then we receive a name like above
        val itemNameAsPackageName = renameContext.itemName.split(".")
        // Should end up as "design.animus.kotlin.mp.schemas.kubernetes.generated.io.k8s.kube-aggregator.pkg.apis.apiregistration.v1beta1"
        "${renameContext.packageName}.${itemNameAsPackageName.subList(0,itemNameAsPackageName.size - 1).joinToString(".")}"
      }
    )
  )
}

This is identical to the prior example with the exception of mutateObjectName and mutatePackageName.

Explaining mutateXYZ

A schema may contain something that is either not compatible with Kotlin naming conventions. Or more commonly demonstarates an internal namespace structure. Kubernetes and AWS are good examples of this.

In the above example we assumae an object name of.

io.k8s.kube-aggregator.pkg.apis.apiregistration.v1beta1.ServiceReference

If we were to use the standard sanitize function, it would come out something like.

IoK8sKubeAggregatorPkgApisApiregistrationV1beta1ServiceReference

That is down right nasty, and would be horrible to type out. Looking at the below screen shot, looks much cleaner. Items fall under the proper name space.

With the mutate function we are given a rename context which is the following. There is another mutate function, thanks Azure (*foreshadowing*).

data class RenameContext(val packageName: String, val itemName: String)
  • packageName Is the target package name at that point in parsing of the schema.
  • itemName is the current class name.

Both functions accept an Option<RenameContextFunction>. Which looks like.

typealias RenameContextFunction = (RenameContext) -> String

For the object name, we are splitting on . and just taking the last element. In the above example it would be ServiceReference.

Conversely for the packageName, we are splitting on . and taking everything but the last item. Which would amend io.k8s.kube-aggregator.pkg.apis.apiregistration.v1beta1 to the package name. Without diving deep into the intrinsic of the JSON schema. Based on the position in the schema the aforementioned packageBase is amended based on the current location in the schema.

Amend the packageBase do not over write.

Generating the code

./gradlew :kubernetes:openAPIGenerate

This is very similar to the json schema plugin. Outputting the generated models into generated/commonMain/kotlin/$packageBase. The difference is unlike json schema there is not a root model. So we need a mapper, to parse two key components kind and apiVersion, to determine what to use. Note: this is a stub until I add vendor support for open api.

More Kubernetes

It just doesn't end. Recall the portion of the root serializer versus nested serializer. With the Kubernetes specification we will not be calling the root serializer, unlike say CloudFormation.

Every kubernetes object starts with the following. For an explanation and introduction see kubernetes api groups.

data class UnknownKubernetesRecord(val apiVersion: String, val kind: String)

At this time I am using a mapper, to say if apiVersion is X, and kind is Y. Then use this serializer. While this is not covering all edge cases. The name space above is indicative of how the apiGroups are layed out. This name space can be parsed out to determine both the apiVersion and kind. Building an exhaustive when clase. Needing to just cast the final result.

val serializer = when (record) {
    UnknownKubernetesRecord(apiVersion="apps/v1", kind="ControllerRevision" ) ->
        ControllerRevisionSerializer
        ...
        UnknownKubernetesRecord(apiVersion="apps/v1", kind="StatefulSetStatus" ) ->
        StatefulSetStatusSerializer
 }
val serializer = determineSerializerForKubernetesRecord(unknown)
val deploymentSerializer = serializer as? DeploymentSerializer ?: error("The serializer was not a Deployment as expected.")
val rawAsJson = convertYAMLtoJSON(raw)
val deployment = Json.decodeFromString(deploymentSerializer, rawAsJson)

Above we determine the serializer necessary for this resource. Convert the yaml resource to json. Then use the above serializer to parse it out the anticipated data class. An error will be thrown if it is not a DeploymentSerializer

Important: The mapper is automatically generated during CI/CD. So if new items are added to the kubernetes api. The when clause will be updated. But I do have to manually add new unit tests to cover new types.

Closing

This was just an initial post on how we can go from json schema to open api to parsing the kubernetes model. All of this dog foods off one another. With the wide support of json schema it provides a lot of options. More coming soon.