Run Scheduled Jobs in K8s Directly from git repo (without CI)

Dishant Pandya
5 min readFeb 11, 2023

--

Photo by Aron Visuals on Unsplash

Demonstrating a Multiple Container Cronjob execution directly from git repo using init Containers.

Overview

When we are deploying microservices based applications, every piece of code doesn’t necessarily needs to be containerized and deployed with certain resources allocated, there are one-off tasks, or scheduled tasks which are small piece of code doing just one thing, at a defined intervals. We may find out that even such tasks which are made of just a single line of database query or few lines of code, are ususally included in the same CI/CD workflows that deploy the core services that make the apis. Its an unnecessary set of steps it has to go through, increasing the time it takes to update and execute the job.

Okay! Let’s talk the Solution

I was struggling to sleep one day and then the idea hit me, how about using initContainer in cronJobs to fetch the task code, install the dependency, and execute it, that will make it much more easy to run the task, since the dependency of a one-off task would be low it can be quicky installed as step in init container. And with shared pvc and bash conditions we can ensure we are not cloning the whole repo again and again when run the job and it would rather run a pull to keep updated with latest changes (Hoping you won’t be commiting malformed code in your main branch). And it was pretty simple to build a k8s template for the same, which I’ll be breaking down into steps for you.

Cloning/Syncing the git repo

This is a simple script that copies and sets permission for ssh keys. That we’ll be mounting, for now we’ll be using ssh mode only.

set -eux;
mkdir -p ~/.ssh;
cat /etc/secret-volume/private > ~/.ssh/id_rsa; chmod 400 ~/.ssh/id_rsa;
cat /etc/secret-volume/public > ~/.ssh/id_rsa.pub; chmod 400 ~/.ssh/id_rsa.pub;
mkdir -p /app;
[ -d /app/repo ] && (cd /app/repo; git pull) || git clone -b $GIT_BRANCH $REPO /app/repo;
cd /app/repo; git checkout $GIT_BRANCH;git pull;

this step as initContainer will look something like following.


- name: git-clone
image: alpine/git
volumeMounts:
- name: secret-volume
readOnly: true
mountPath: "/etc/secret-volume"
- name: shared-volume
mountPath: "/app"
env:
- name: REPO
valueFrom:
configMapKeyRef:
name: cron-config
key: repo
- name: GIT_BRANCH
valueFrom:
configMapKeyRef:
name: cron-config
key: gitBranch
- name: GIT_SSH_COMMAND
valueFrom:
configMapKeyRef:
name: cron-config
key: gitSSHCommand
command:
- /bin/sh
- -c
- |
set -eux;
mkdir -p ~/.ssh;
cat /etc/secret-volume/private > ~/.ssh/id_rsa; chmod 400 ~/.ssh/id_rsa;
cat /etc/secret-volume/public > ~/.ssh/id_rsa.pub; chmod 400 ~/.ssh/id_rsa.pub;
mkdir -p /app;
[ -d /app/repo ] && (cd /app/repo; git pull) || git clone -b $GIT_BRANCH $REPO /app/repo;
cd /app/repo; git checkout $GIT_BRANCH;git pull;
resources:
requests:
memory: 200Mi

Installing Dependencies:

In this demo we are using node hence we’ll be running npm install to install the node_modules. The initContainer would look like.

- name: prepare
image: node:16-alpine
volumeMounts:
- name: secret-volume
readOnly: true
mountPath: "/etc/secret-volume"
- name: shared-volume
mountPath: "/app"
env:
- name: NODE_TLS_REJECT_UNAUTHORIZED
value: "0"
- name: REPO
valueFrom:
configMapKeyRef:
name: cron-config
key: repo
- name: TASK_PATH
valueFrom:
configMapKeyRef:
name: cron-config
key: taskPath
command:
- /bin/sh
- -c
- |
set -eux;
cd /app/repo/$TASK_PATH;
yarn config set "strict-ssl" false;
yarn;
resources:
requests:
memory: 1024Mi

Configuring the Primary Container:

Once our clone and prepare initContainers are defined we can define the Container.

It will use the TASK_FILE variable to define the executable file, and we’ll be using node runtime to execute it, it can be configured to fit your language of choice.

containers:
- name: taskrun
image: node:16-alpine
imagePullPolicy: IfNotPresent
volumeMounts:
- name: shared-volume
mountPath: "/app"
envFrom:
- secretRef:
name: backend
env:
- name: REPO
valueFrom:
configMapKeyRef:
name: cron-config
key: repo
- name: TASK_PATH
valueFrom:
configMapKeyRef:
name: cron-config
key: taskPath
- name: TASK_FILE
valueFrom:
configMapKeyRef:
name: cron-config
key: taskFile
command:
- /bin/sh
- -c
- |
set -eux;
date; ls -l /app/repo/$TASK_PATH;
node /app/repo/$TASK_PATH/$TASK_FILE;
sleep 5;
readinessProbe:
exec:
command:
- ls
- /app/repo
initialDelaySeconds: 600
periodSeconds: 10

The whole cronjob will look like.

apiVersion: batch/v1
kind: CronJob
metadata:
name: cron
labels:
app.kubernetes.io/component: cron
spec:
schedule: "* * * * *"
concurrencyPolicy: Forbid
startingDeadlineSeconds: 30
jobTemplate:
spec:
template:
spec:
volumes:
- name: secret-volume
secret:
secretName: cron-git-creds
- name: shared-volume
persistentVolumeClaim:
claimName: cron
securityContext:
runAsUser: 0
runAsGroup: 0
fsGroup: 0
terminationGracePeriod: 10
initContainers:
- name: git-clone
image: alpine/git
volumeMounts:
- name: secret-volume
readOnly: true
mountPath: "/etc/secret-volume"
- name: shared-volume
mountPath: "/app"
env:
- name: REPO
valueFrom:
configMapKeyRef:
name: cron-config
key: repo
- name: GIT_BRANCH
valueFrom:
configMapKeyRef:
name: cron-config
key: gitBranch
- name: GIT_SSH_COMMAND
valueFrom:
configMapKeyRef:
name: cron-config
key: gitSSHCommand
command:
- /bin/sh
- -c
- |
set -eux;
mkdir -p ~/.ssh;
cat /etc/secret-volume/private > ~/.ssh/id_rsa; chmod 400 ~/.ssh/id_rsa;
cat /etc/secret-volume/public > ~/.ssh/id_rsa.pub; chmod 400 ~/.ssh/id_rsa.pub;
mkdir -p /app;
[ -d /app/repo ] && (cd /app/repo; git pull) || git clone -b $GIT_BRANCH $REPO /app/repo;
cd /app/repo; git checkout $GIT_BRANCH;git pull;
resources:
requests:
memory: 200Mi
- name: prepare
image: node:16-alpine
volumeMounts:
- name: secret-volume
readOnly: true
mountPath: "/etc/secret-volume"
- name: shared-volume
mountPath: "/app"
env:
- name: NODE_TLS_REJECT_UNAUTHORIZED
value: "0"
- name: REPO
valueFrom:
configMapKeyRef:
name: cron-config
key: repo
- name: TASK_PATH
valueFrom:
configMapKeyRef:
name: cron-config
key: taskPath
command:
- /bin/sh
- -c
- |
set -eux;
cd /app/repo/$TASK_PATH;
yarn config set "strict-ssl" false;
yarn;
resources:
requests:
memory: 1024Mi
containers:
- name: taskrun
image: node:16-alpine
imagePullPolicy: IfNotPresent
volumeMounts:
- name: shared-volume
mountPath: "/app"
envFrom:
- secretRef:
name: backend
env:
- name: REPO
valueFrom:
configMapKeyRef:
name: cron-config
key: repo
- name: TASK_PATH
valueFrom:
configMapKeyRef:
name: cron-config
key: taskPath
- name: TASK_FILE
valueFrom:
configMapKeyRef:
name: cron-config
key: taskFile
command:
- /bin/sh
- -c
- |
set -eux;
date; ls -l /app/repo/$TASK_PATH;
node /app/repo/$TASK_PATH/$TASK_FILE;
sleep 5;
readinessProbe:
exec:
command:
- ls
- /app/repo
initialDelaySeconds: 600
periodSeconds: 10
restartPolicy: OnFailure

SSH Secret

---
apiVersion: v1
data:
public: <base64-public-key>
private: <base64-private-key>
kind: Secret
metadata:
name: cron-git-creds

Persistent Volume Claim

This is required to persist the repo and installed dependencies, the init container will check and pull the latest code from the repo everytime the cron is run. Reducing the simultaneous execution times.

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cron
spec:
resources:
requests:
storage: 200Mi
accessModes:
- ReadWriteOnce

Example Config

Following is the config with working example from a repo.

apiVersion: v1
kind: ConfigMap
metadata:
name: cron-config
data:
repo: "git@github.com:drpdishant/sample-node-app.git"
taskPath: "" #Path WRT repo root
taskFile: "tasks/task-a.js" #Name of the file to be executed as task
gitSSHCommand: "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"
gitBranch: "develop"

The config parameters are as follows:

repo: ssh url of the git repo
taskPath: Path to directory were task files are located wrt repo root
taskFile: Path to task executable wrt task path.
gitSSHCommand: ssh options to be configured for git
gitBranch: Git branch to checkout

Conclusion

In this demo we learned with example how we can directly run tasks in CronJobs using git using initContainer to pull/clone the repo and install dependencies. This can be extended with additional configurable parameters for different languages and templating it with helm to provide a generic cron executor that runs tasks directly from git.

Please share your comments and do let me know if this helps you.

--

--

Dishant Pandya

Platform Engineer @Kotak811. Building handcrafted solutions to meet high velocity application development.