Run Scheduled Jobs in K8s Directly from git repo (without CI)

Photo by Aron Visuals on Unsplash

Demonstrating a Multiple Container Cronjob execution directly from git repo using init Containers.

Overview

When we are deploying microservices based applications, every piece of code doesn’t necessarily needs to be containerized and deployed with certain resources allocated, there are one-off tasks, or scheduled tasks which are small piece of code doing just one thing, at a defined intervals. We may find out that even such tasks which are made of just a single line of database query or few lines of code, are ususally included in the same CI/CD workflows that deploy the core services that make the apis. Its an unnecessary set of steps it has to go through, increasing the time it takes to update and execute the job.

Okay! Let’s talk the Solution

I was struggling to sleep one day and then the idea hit me, how about using initContainer in cronJobs to fetch the task code, install the dependency, and execute it, that will make it much more easy to run the task, since the dependency of a one-off task would be low it can be quicky installed as step in init container. And with shared pvc and bash conditions we can ensure we are not cloning the whole repo again and again when run the job and it would rather run a pull to keep updated with latest changes (Hoping you won’t be commiting malformed code in your main branch). And it was pretty simple to build a k8s template for the same, which I’ll be breaking down into steps for you.

Cloning/Syncing the git repo

This is a simple script that copies and sets permission for ssh keys. That we’ll be mounting, for now we’ll be using ssh mode only.

set -eux;
mkdir -p ~/.ssh;
cat /etc/secret-volume/private > ~/.ssh/id_rsa; chmod 400 ~/.ssh/id_rsa;
cat /etc/secret-volume/public > ~/.ssh/id_rsa.pub; chmod 400 ~/.ssh/id_rsa.pub;
mkdir -p /app;
[ -d /app/repo ] && (cd /app/repo; git pull) || git clone -b $GIT_BRANCH $REPO /app/repo;
cd /app/repo; git checkout $GIT_BRANCH;git pull;

this step as initContainer will look something like following.

- name: git-clone
  image: alpine/git
  volumeMounts:
  - name: secret-volume
    readOnly: true
    mountPath: "/etc/secret-volume"
  - name: shared-volume
    mountPath: "/app"
  env:
  - name: REPO 
    valueFrom:
      configMapKeyRef:
        name: cron-config          
        key: repo
  - name: GIT_BRANCH
    valueFrom:
      configMapKeyRef:
        name: cron-config
        key: gitBranch
  - name: GIT_SSH_COMMAND
    valueFrom:
      configMapKeyRef:
        name: cron-config
        key: gitSSHCommand
  command:
  - /bin/sh
  - -c
  - |
    set -eux;
    mkdir -p ~/.ssh;
    cat /etc/secret-volume/private > ~/.ssh/id_rsa; chmod 400 ~/.ssh/id_rsa;
    cat /etc/secret-volume/public > ~/.ssh/id_rsa.pub; chmod 400 ~/.ssh/id_rsa.pub;
    mkdir -p /app;
    [ -d /app/repo ] && (cd /app/repo; git pull) || git clone -b $GIT_BRANCH $REPO /app/repo;
    cd /app/repo; git checkout $GIT_BRANCH;git pull;
  resources:
    requests: 
      memory: 200Mi

Installing Dependencies:

In this demo we are using node hence we’ll be running npm install to install the node_modules. The initContainer would look like.

- name: prepare
  image: node:16-alpine
  volumeMounts:
  - name: secret-volume
    readOnly: true
    mountPath: "/etc/secret-volume"
  - name: shared-volume
    mountPath: "/app"
  env: 
  - name: NODE_TLS_REJECT_UNAUTHORIZED
    value: "0"
  - name: REPO 
    valueFrom:
      configMapKeyRef:
        name: cron-config          
        key: repo 
  - name: TASK_PATH
    valueFrom:
      configMapKeyRef:
        name: cron-config
        key: taskPath
  command:
  - /bin/sh
  - -c
  - |
    set -eux;
    cd /app/repo/$TASK_PATH;
    yarn config set "strict-ssl" false;
    yarn;
  resources:
    requests: 
      memory: 1024Mi
  

Configuring the Primary Container:

Once our clone and prepare initContainers are defined we can define the Container.

It will use the TASK_FILE variable to define the executable file, and we’ll be using node runtime to execute it, it can be configured to fit your language of choice.

containers:
- name: taskrun
  image: node:16-alpine
  imagePullPolicy: IfNotPresent
  volumeMounts:
    - name: shared-volume
      mountPath: "/app"
  envFrom:
  - secretRef:
      name: backend
  env: 
    - name: REPO 
      valueFrom:
        configMapKeyRef:
          name: cron-config          
          key: repo 
    - name: TASK_PATH
      valueFrom:
        configMapKeyRef:
          name: cron-config
          key: taskPath
    - name: TASK_FILE
      valueFrom:
        configMapKeyRef:
          name: cron-config
          key: taskFile
  command:
  - /bin/sh
  - -c
  - |
    set -eux;
    date; ls -l /app/repo/$TASK_PATH;
    node /app/repo/$TASK_PATH/$TASK_FILE;
    sleep 5;
  readinessProbe:
      exec:
        command:
        - ls
        - /app/repo
      initialDelaySeconds: 600
      periodSeconds: 10

The whole cronjob will look like.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: cron
  labels:
    app.kubernetes.io/component: cron
spec:
  schedule: "* * * * *"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 30
  jobTemplate:
    spec:
      template:
        spec:
          volumes:
          - name: secret-volume
            secret:
              secretName: cron-git-creds
          - name: shared-volume
            persistentVolumeClaim:
              claimName: cron
          securityContext:
            runAsUser: 0
            runAsGroup: 0
            fsGroup: 0
          terminationGracePeriod: 10
          initContainers:
            - name: git-clone
              image: alpine/git
              volumeMounts:
              - name: secret-volume
                readOnly: true
                mountPath: "/etc/secret-volume"
              - name: shared-volume
                mountPath: "/app"
              env:
              - name: REPO 
                valueFrom:
                  configMapKeyRef:
                    name: cron-config          
                    key: repo 
              - name: GIT_BRANCH
                valueFrom:
                  configMapKeyRef:
                    name: cron-config
                    key: gitBranch
              - name: GIT_SSH_COMMAND
                valueFrom:
                  configMapKeyRef:
                    name: cron-config
                    key: gitSSHCommand
              command:
              - /bin/sh
              - -c
              - |
                set -eux;
                mkdir -p ~/.ssh;
                cat /etc/secret-volume/private > ~/.ssh/id_rsa; chmod 400 ~/.ssh/id_rsa;
                cat /etc/secret-volume/public > ~/.ssh/id_rsa.pub; chmod 400 ~/.ssh/id_rsa.pub;
                mkdir -p /app;
                [ -d /app/repo ] && (cd /app/repo; git pull) || git clone -b $GIT_BRANCH $REPO /app/repo;
                cd /app/repo; git checkout $GIT_BRANCH;git pull;
              resources:
                requests: 
                  memory: 200Mi
            - name: prepare
              image: node:16-alpine
              volumeMounts:
              - name: secret-volume
                readOnly: true
                mountPath: "/etc/secret-volume"
              - name: shared-volume
                mountPath: "/app"
              env: 
              - name: NODE_TLS_REJECT_UNAUTHORIZED
                value: "0"
              - name: REPO 
                valueFrom:
                  configMapKeyRef:
                    name: cron-config          
                    key: repo 
              - name: TASK_PATH
                valueFrom:
                  configMapKeyRef:
                    name: cron-config
                    key: taskPath
              command:
              - /bin/sh
              - -c
              - |
                set -eux;
                cd /app/repo/$TASK_PATH;
                yarn config set "strict-ssl" false;
                yarn;
              resources:
                requests: 
                  memory: 1024Mi
          containers:
          - name: taskrun
            image: node:16-alpine
            imagePullPolicy: IfNotPresent
            volumeMounts:
              - name: shared-volume
                mountPath: "/app"
            envFrom:
            - secretRef:
                name: backend
            env: 
              - name: REPO 
                valueFrom:
                  configMapKeyRef:
                    name: cron-config          
                    key: repo 
              - name: TASK_PATH
                valueFrom:
                  configMapKeyRef:
                    name: cron-config
                    key: taskPath
              - name: TASK_FILE
                valueFrom:
                  configMapKeyRef:
                    name: cron-config
                    key: taskFile
            command:
            - /bin/sh
            - -c
            - |
              set -eux;
              date; ls -l /app/repo/$TASK_PATH;
              node /app/repo/$TASK_PATH/$TASK_FILE;
              sleep 5;
            readinessProbe:
                exec:
                  command:
                  - ls
                  - /app/repo
                initialDelaySeconds: 600
                periodSeconds: 10
          restartPolicy: OnFailure

SSH Secret

---
apiVersion: v1
data:
  public: <base64-public-key>
  private: <base64-private-key>
kind: Secret
metadata:
  name: cron-git-creds

Persistent Volume Claim

This is required to persist the repo and installed dependencies, the init container will check and pull the latest code from the repo everytime the cron is run. Reducing the simultaneous execution times.

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cron
spec:
  resources:
    requests:
      storage: 200Mi
  accessModes: 
    - ReadWriteOnce

Example Config

Following is the config with working example from a repo.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cron-config
data:
  repo: "git@github.com:drpdishant/sample-node-app.git"
  taskPath: "" #Path WRT repo root
  taskFile: "tasks/task-a.js" #Name of the file to be executed as task
  gitSSHCommand: "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"
  gitBranch: "develop"

The config parameters are as follows:

repo: ssh url of the git repo
taskPath: Path to directory were task files are located wrt repo root
taskFile: Path to task executable wrt task path.
gitSSHCommand: ssh options to be configured for git
gitBranch: Git branch to checkout

Conclusion

In this demo we learned with example how we can directly run tasks in CronJobs using git using initContainer to pull/clone the repo and install dependencies. This can be extended with additional configurable parameters for different languages and templating it with helm to provide a generic cron executor that runs tasks directly from git.

Please share your comments and do let me know if this helps you.