Runnable: Templating Objects That Cannot Update

Overview

In this tutorial we’ll explore a new Cartographer resource: runnable. Runnables will enable us to choreograph resources that do not support standard update behavior. We’ll see how we can wrap Runnables around tools useful for testing, like Tekton, and how that will provide us easy updating behavior of our testing objects.

Environment setup

For this tutorial you will need a kubernetes cluster with Cartographer and Tekton installed. You can find Cartographer’s installation instructions here and Tekton’s installation instructions are here.

Alternatively, you may choose to use the ./hack/setup.sh script to install a kind cluster with Cartographer and Tekton. This script is meant for our end-to-end testing and while we rely on it working in that role, no user guarantees are made about the script.

Command to run from the Cartographer directory:

./hack/setup.sh cluster cartographer-latest example-dependencies

If you later wish to tear down this generated cluster, run

./hack/setup.sh teardown

Scenario

Our CTO is interested in putting quality controls in place; only code that passes certain checks should be built and deployed. They want to start small: all source code repositories that are built must pass markdown linting. In order to do this we’re going to leverage the markdown linting pipeline in the TektonCD catalog.

In this tutorial we’ll see how to use Cartographer’s Runnable to give us easy updating behavior of Tekton (no need for Tekton Triggers and Github Webhooks). In the next tutorial we’ll complete the scenario by using Runnable in a supply chain.

Steps

Tekton Basics

Before using Cartographer, let’s think about how we would use Tekton on its own to lint a repo. First we would define a pipeline:

---
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: linter-pipeline
spec:
  params:
    - name: repository
      type: string
    - name: revision
      type: string
  workspaces:
    - name: shared-workspace
  tasks:
    - name: fetch-repository
      taskRef:
        name: git-clone
      workspaces:
        - name: output
          workspace: shared-workspace
      params:
        - name: url
          value: $(params.repository)
        - name: revision
          value: $(params.revision)
        - name: subdirectory
          value: ""
        - name: deleteExisting
          value: "true"
    - name: md-lint-run #lint markdown
      taskRef:
        name: markdown-lint
      runAfter:
        - fetch-repository
      workspaces:
        - name: shared-workspace
          workspace: shared-workspace
      params:
        - name: args
          value: ["."]

We would apply this pipeline to the cluster, along with the tasks. Those tasks are in the TektonCD Catalog:

kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/git-clone/0.3/git-clone.yaml
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/markdown-lint/0.1/markdown-lint.yaml

Finally, we need to create a pipeline-run object. This object provides the param and workspace values defined at the top level .spec field of the pipeline.

---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: linter-pipeline-run
spec:
  pipelineRef:
    name: linter-pipeline
  params:
    - name: repository
      value: https://github.com/waciumawanjohi/demo-hello-world
    - name: revision
      value: main
  workspaces:
    - name: shared-workspace
      volumeClaimTemplate:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 256Mi

Importantly, this pipeline-run object will kick off a single run of the pipeline. The run will either succeed or fail. The outcome will be written into the pipeline-run’s status. No later changes to the pipeline-run object will change those outcomes; the run happens once.

To see this in action, let’s apply the above pipeline-run. If we watch the object, we’ll soon see that it succeeds.

watch 'kubectl get -o yaml pipelinerun linter-pipeline-run | yq .status.conditions'

Eventually yields the result:

- lastTransitionTime: ...
  message: "Tasks Completed: 2 (Failed: 0, Cancelled 0), Skipped: 0"
  reason: "Succeeded"
  status: "True"
  type: "Succeeded"

Templating Pipeline Runs

This seems like a very easy step to encode in a supply chain. All we would need to do is ensure that the Tekton tasks and pipeline are created beforehand. Then a supply chain could stamp out a templated pipeline-run object. This template will pull the repository and revision value from the workload. But there’s a problem… what happens if an app dev changes one of these values on the workload? Our supply chain would not properly reflect the change, because the pipeline-run object cannot be updated!

Fortunately there’s an easy fix. We’re going to use a pair of new Cartographer resources: Runnable and ClusterRunTemplate.

It will be the responsibility of the ClusterRunTemplate to template our desired object (in this case, our Tekton pipeline-run). The Runnable will be responsible for providing values to the ClusterRunTemplate, the values that we expect to vary (in our example, the url and revision of the source code). When the set of values from the Runnable changes, a new object will be created (rather than the old object updated).

The Runnable will also expose results. Of course, multiple results will exist, a result for each of the objects created. Runnable will only expose the results from the most recently submitted successful object.

In this manner, we get a wrapper (Runnable) that is updateable and updates results in its status. This is similar to the default behavior of kuberenetes objects. By wrapping Tekton pipeline-runs (or any immutable resource) in a Runnable, we will be able to use the resource in a supply chain as if it were mutable.

Let’s see the Runnable and the ClusterRunTemplate at work. Once we’re solid on those, we’ll use Runnable in a Supply Chain in the next tutorial.

ClusterRunTemplate

Let’s start with the ClusterRunTemplate. As can be expected from the name, there’s a .spec.template field in it, where we will write something very similar to our pipeline-run above. In fact, let’s write that exact pipeline-run in and then look at the values that will need to change:

apiVersion: carto.run/v1alpha1
kind: ClusterRunTemplate
metadata:
  name: md-linting-pipelinerun
spec:
  template:
    apiVersion: tekton.dev/v1beta1
    kind: PipelineRun
    metadata:
      name: linter-pipeline-run # <=== Can’t all have the same name
    spec:
      pipelineRef:
        name: linter-pipeline
      params: # <=== These param values will change
        - name: repository
          value: https://github.com/waciumawanjohi/demo-hello-world
        - name: revision
          value: main
      workspaces:
        - name: shared-workspace
          volumeClaimTemplate:
            spec:
              accessModes:
                - ReadWriteOnce
              resources:
                requests:
                  storage: 256Mi

Most fields are fine. The name field is not. Why not? If we want to change the values for the pipeline-run, we’re not going to update the templated object. We’re going to create an entirely new object. And of course that new object can’t have the same hardcoded name. To handle this, every object templated in a ClusterRunTemplate specifies a generateName rather than a name. We can use linter-pipeline-run- and kubernetes will handle putting a unique suffix on the name of each pipeline-run.

    metadata:
      generateName: linter-pipeline-run-

The other change we want to make is to the values on the params. It doesn’t do much good to hardcode https://github.com/waciumawanjohi/demo-hello-world into the repository param; this is the value we want to update. As we said, we intend to update the Runnable and have the value in that update stamped out in a new pipeline-run object. So we replace the hardcoded value with a Cartographer parameter. We’ll find the value that we want on the runnable. That will look like $(runnable.spec.inputs.repository)$. This specifies that the value we want will be found in the runnable spec, in a field named inputs, one of which will have the name repository.

There’s only one more thing that we need to specify on the ClusterRunTemplate: what the outputs will be! We said that the Runnable status will reflect results of a successful run. The ClusterRunTemplate specifies what results. For now let’s simply report the lastTransition time that we saw on the conditions. We use jsonpath to indicate the location of this value on the objects that are being stamped:

  outputs:
    lastTransitionTime: .status.conditions[0].lastTransitionTime

Let’s look at our complete ClusterRunTemplate:

---
apiVersion: carto.run/v1alpha1
kind: ClusterRunTemplate
metadata:
  name: md-linting-pipelinerun
spec:
  template:
    apiVersion: tekton.dev/v1beta1
    kind: PipelineRun
    metadata:
      generateName: linter-pipeline-run-
    spec:
      pipelineRef:
        name: linter-pipeline
      params:
        - name: repository
          value: $(runnable.spec.inputs.repository)$
        - name: revision
          value: $(runnable.spec.inputs.revision)$
      workspaces:
        - name: shared-workspace
          volumeClaimTemplate:
            spec:
              accessModes:
                - ReadWriteOnce
              resources:
                requests:
                  storage: 256Mi
  outputs:
    lastTransitionTime: .status.conditions[0].lastTransitionTime

Runnable

Now let’s make the Runnable. First we’ll specify which ClusterRunTemplate our Runnable works with. We do this in the Runnable’s .spec.runTemplateRef. field and we refer to the name of the ClusterRunTemplate we just created.

  runTemplateRef:
    name: md-linting-pipelinerun

Next we’ll fill in the inputs. These are the paths that we templated into the ClusterRunTemplate param values, the runnable.spec.inputs.repository and runnable.spec.inputs.revision. We’ll fill these with the values we previously hardcoded in the pipelinerun. With runnable we’ll be able to update them later as we like.

  inputs:
    repository: https://github.com/waciumawanjohi/demo-hello-world
    revision: main

Finally, we need a serviceAccountName. Just like the supply-chain, with Runnable Cartographer could be stamping out anything. Using RBAC we expect a service account to provide permissions to Cartographer and limit it to creating only the types of objects we expect. We’ll create a service account named pipeline-run-management-sa. We’ll put that name in our Runnable object. The full object looks like this:

---
apiVersion: carto.run/v1alpha1
kind: Runnable
metadata:
  name: linter
spec:
  runTemplateRef:
    name: md-linting-pipelinerun
  inputs:
    repository: https://github.com/waciumawanjohi/demo-hello-world
    revision: main
  serviceAccountName: pipeline-run-management-sa

Let’s quickly create the service account we referenced:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: pipeline-run-management-sa

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: testing-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pipeline-run-management-role
subjects:
  - kind: ServiceAccount
    name: pipeline-run-management-sa

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pipeline-run-management-role
rules:
  - apiGroups:
      - tekton.dev
    resources:
      - pipelineruns
    verbs:
      - list
      - create
      - update
      - delete
      - patch
      - watch
      - get

Great! Let’s deploy these objects.

Observe

Let’s observe the pipeline-run objects in the cluster:

kubectl get pipelineruns

We can see that a new pipelinerun has been created with the linter-pipeline-run- prefix:

NAME                        SUCCEEDED   REASON      STARTTIME   COMPLETIONTIME
linter-pipeline-run-123az   True        Succeeded   2m48s       2m35s

Examining the created object it’s a non-trivial 300 lines:

kubectl get -o yaml pipelineruns linter-pipeline-run-123az

In the metadata we can see familiar labels indicating Carto objects used to create this templated object. We can also see that the object is owned by the runnable.

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: linter-pipeline-run-123az
  generateName: linter-pipeline-run-
  labels:
    carto.run/run-template-name: md-linting-pipelinerun
    carto.run/runnable-name: linter
    tekton.dev/pipeline: linter-pipeline
  ownerReferences:
    - apiVersion: carto.run/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Runnable
      name: linter
      uid: ...
  ...

The spec contains the spec that we templated out. Looks great.

spec:
  params:
    - name: repository
      value: https://github.com/waciumawanjohi/demo-hello-world
    - name: revision
      value: main
  pipelineRef:
    name: linter-pipeline
  serviceAccountName: default
  timeout: 1h0m0s
  workspaces:
    - name: shared-workspace
      volumeClaimTemplate:
        metadata:
          creationTimestamp: null
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 256Mi
        status: {}

The status contains fields expected of Tekton:

status:
  completionTime: ...
  conditions:
    - lastTransitionTime: "2022-03-07T19:24:35Z"
      message: "Tasks Completed: 2 (Failed: 0, Cancelled 0), Skipped: 0"
      reason: Succeeded
      status: "True"
      type: Succeeded
  pipelineSpec: ...
  startTime: ...
  taskRuns: ...

To learn more about Tekton’s behavior, readers will want to refer to Tekton documentation.

Now we examine the Cartographer Runnable object. We expect it to expose values from our successful object.

kubectl get -o yaml runnable linter

apiVersion: carto.run/v1alpha1
kind: Runnable
Metadata: ...
Spec: ...
status:
  conditions: ...
  observedGeneration: ...
  outputs:
    lastTransitionTime: "2022-03-07T19:24:35Z"

Wonderful! The value from the field we specified in the ClusterRunTemplate is now in the outputs of the Runnable.

Finally, let’s update our runnable with a new repository:

apiVersion: carto.run/v1alpha1
kind: Runnable
metadata:
  name: linter
spec:
  runTemplateRef:
    name: md-linting-pipelinerun
  inputs:
    repository: https://github.com/kelseyhightower/nocode # <=== new repo
    revision: main # <=== new revision
  serviceAccountName: pipeline-run-management-sa

When we apply this to the cluster, we can observe:

• The spec of our runnable is updated
• The runnable causes the creation of a new pipelinerun
• The new pipeline run fails because the new repo does not pass linting
• Since the result of the new pipeline run is failure, our runnable output remains the output of our previous (successful) pipeline run

Wrap Up

In this tutorial you learned:

• Some useful kubernetes objects cannot be updated
• Runnable allows you to add updateable behavior to those objects
• How to write Runnables with input values for ClusterRunTemplates
• How to write ClusterRunTemplates that specify outputs for the Runnable
• How to read output values from the Runnable

We’ve got an updateable object, Runnable, that can manage tekton pipelines and tasks. Our next step is going to be using Runnable in a supply chain.