As you start your container journey, it is mandatory that you have image build tools. There are various tools available; the most common ones are Docker, Buildah, Podman, and Kaniko. You need to have image build tools to pull down images from external registry, customize the images, and build new images with libraries or applications. These images then published to container registries and further used in Kubernetes platforms such as Red Hat OpenShift, EKS, AKS, etc.
In this article, we'll be mainly talking about OpenShift, but the contents can be used for other platforms as well with little to no modifications.
In most enterprises, especially where security, compliance, and operational consistency are paramount, image build processes are strictly governed due to control risks. I have seen organizations having a separate build infrastructure just to manage the image build process. On the other side, few organizations allow application teams to manage build infrastructure. In both the cases, the infrastructure needs to have any of the above listed tools available. It is very common that such tools are mostly installed on virtual machines (VMs) reserved just for container build processes. Along with such infrastructure, it also comes with the overhead of maintenance, security, capacity management, and more.
Why we are talking about just container image builds? It is because most of the other pipeline stages such as application compilation or build, packaging, and deployment can be handled within containers on OpenShift. Only image build process needs specific configurations to make it work.
If you are using GitLab as CI/CD tool, you have to manage various GitLab runners (meaning various virtual machines) with approved container image build tools. If your organization has OpenShift, then Podman would be the tool of choice. In such scenarios, if we move these runners on OpenShift, wouldn't that be cost beneficial if we don't have to manage and maintain VMs just for image builds anymore?
Let's look at what is required to run Podman on OpenShift that can enable the image build process in OpenShift itself.
Requirements to run Podman on OpenShift
Note
In order to perform some of the following steps, you should have admin access in OpenShift.
Create new Security Context Constraint (SCC) in OpenShift.
OpenShift 4.11 introduced
restricted-v2SCC; however, prior to that the default SCC wasrestrictedSCC. Create a new custom SCC based on the restricted SCC by adding belowSETUIDandSETGIDcapabilities. Below is the complete SCC that should be created:apiVersion: security.openshift.io/v1 metadata: name: podman-scc allowHostDirVolumePlugin: false allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: true allowPrivilegedContainer: false allowedCapabilities: - SETUID - SETGID defaultAddCapabilities: null fsGroup: type: MustRunAs groups: [] kind: SecurityContextConstraints priority: null readOnlyRootFilesystem: false requiredDropCapabilities: - KILL - MKNOD runAsUser: type: MustRunAsRange seLinuxContext: type: MustRunAs supplementalGroups: type: RunAsAny users: [] volumes: - configMap - downwardAPI - emptyDir - persistentVolumeClaim - projected - secretCreate
entrypoint.shto use in the Containerfile:#!/usr/bin/env bash if [ ! -d "${HOME}" ] then mkdir -p "${HOME}" fi mkdir -p ${HOME}/.config/containers # Below line is not needed for ocp 4.15+ clusters as fuse-overlayfs will be made available (echo '[storage]';echo 'driver = "vfs"') > ${HOME}/.config/containers/storage.conf if ! whoami &> /dev/null then if [ -w /etc/passwd ] then echo "${USER_NAME:-user}:x:$(id -u):0:${USER_NAME:-user} user:${HOME}:/bin/bash" >> /etc/passwd echo "${USER_NAME:-user}:x:$(id -u):" >> /etc/group fi fi USER=$(whoami) START_ID=$(( $(id -u)+1 )) echo "${USER}:${START_ID}:2147483646" > /etc/subuid echo "${USER}:${START_ID}:2147483646" > /etc/subgid exec "$@"Create the container image with Podman and additional configurations to run Podman in rootless mode.
If you have ever tried to run the Podman locally or on a VM in the rootless mode, you should be familiar with some of the configurations below in the Containerfile (Dockerfile). The pod in OpenShift comes up with a default user with a random user ID (UID). To set up the required user permissions to run Podman commands, you need to make the following configurations. Build the image with the below Containerfile and publish it to image repository. The below Containerfile also installs other tools along with Podman but it is not required:
FROM registry.access.redhat.com/ubi9/ubi-minimal ARG USER_HOME_DIR="/home/user" ENV HOME=${USER_HOME_DIR} ENV BUILDAH_ISOLATION=chroot RUN microdnf --disableplugin=subscription-manager install -y openssl git tar which shadow-utils bash zsh wget jq podman buildah skopeo; \ microdnf update -y ; \ microdnf clean all ; \ mkdir -p ${USER_HOME_DIR} ; \ chgrp -R 0 /home ; \ # # Setup for root-less podman # setcap cap_setuid+ep /usr/bin/newuidmap ; \ setcap cap_setgid+ep /usr/bin/newgidmap ; \ touch /etc/subgid /etc/subuid ; \ chmod -R g=u /etc/passwd /etc/group /etc/subuid /etc/subgid /home WORKDIR ${HOME} COPY entrypoint.sh . RUN chmod +x entrypoint.sh ENTRYPOINT [ "./entrypoint.sh" ]Create a namespace in OpenShift for running the GitLab runner instance, e.g.,
podman-runner:oc create ns podman-runnerCreate service account in the above namespace and associate the SCC with the service account:
oc -n podman-runner create sa podman-sa oc -n podman-runner adm policy add-scc-to-user podman-scc -z podman-saCreate
imagePullSecretto access the image built in step 3 from external registry. ThisimagePullSecretis referenced in the subsequent step for deployment resource:kind: Secret apiVersion: v1 metadata: name: gitlab-secret namespace: podman-runner data: .dockerconfigjson: eyJhdXRocyI6eIiwicGFzc3dvcmQiOiJAbGVhcm5HaXRsYWIwIiwiYXV0aCI6IllURnFhMkZ1T2tCc1pXRnlia2RwZEd4aFlqQT0iLCJlbWFpbCI6IiJ9fX0= type: kubernetes.io/dockerconfigjsonCreate an example deployment in the namespace to test the Podman image and the functionality:
# Created podman-runner namespace kind: Deployment apiVersion: apps/v1 metadata: name: podman-example namespace: podman-runner spec: replicas: 1 selector: matchLabels: app: name template: metadata: labels: app: name spec: containers: - name: container # Below image is built at step 3 and published to gitlab container repository image: 'registry.gitlab.com/myrepo/podman-runner:1.0.0' # Command with below added so that container doesn't go away by the time we complete test command: - /bin/sh - '-c' - ./entrypoint.sh sleep 3600 imagePullPolicy: Always securityContext: allowPrivilegeEscalation: true capabilities: add: - "SETUID" - "SETGID" serviceAccountName: podman-sa restartPolicy: Always imagePullSecrets: - name: gitlab-secretValidate that Podman commands can be executed in the
container terminalin OpenShift.Make sure configurations done in the entrypoint script (step 2) for OpenShift random ID are in effect correctly:
# Below commands are executed in the terminal window of container for validation # Note that random uid 1000850000 has necessary configurations for this container sh-5.1$ podman version Client: Podman Engine Version: 4.9.4-rhel API Version: 4.9.4-rhel Go Version: go1.21.7 (Red Hat 1.21.7-1.el9) Built: Mon Apr 15 12:33:08 2024 OS/Arch: linux/amd64 sh-5.1$ cat /etc/sub{u,g}id ; cat /etc/passwd | grep user ; cat /etc/group | grep user ; getcap /usr/bin/newuidmap ; getcap /usr/bin/newgidmap ; podman info | grep -i driver user:1000850001:2147483646 user:1000850001:2147483646 user:x:1000850000:0:user user:/home/user:/bin/bash users:x:100: user:x:1000850000: /usr/bin/newuidmap cap_setuid=ep /usr/bin/newgidmap cap_setgid=ep logDriver: k8s-file graphDriverName: vfs
Set up and configure GitLab runner
Now that the Podman container is validated, let's set up a GitLab runner to use the image that was built in step 3. The steps below provide high-level details on how to set up and configure the GitLab runner.
Install the GitLab Runner operator labeled Certified from OperatorHub in the
podman-runnernamespace, as shown in Figure 1.
Figure 1: Gitlab runner operator. - In the GitLab project that you intend to build, create a Kubernetes runner and get the
runner authentication token. Create Kubernetes secret in the
podman-runnernamespace with the runner authentication token:kind: Secret apiVersion: v1 metadata: name: gitlab-runner-secret namespace: podman-runner data: runner-registration-token: Z2xydC1CY1d4UjNvdw==Create the ConfigMap with
config.tomlkey for GitLab runner configurations:kind: ConfigMap apiVersion: v1 metadata: name: podman-gitlab-runner-custom-toml namespace: podman-runner data: config.toml: | [[runners]] [runners.kubernetes] # Below overwrite configurations are required for podman runner pod_annotations_overwrite_allowed = "openshift.io/required-scc=.*" service_account_overwrite_allowed = ".*" pull_policy = "always" # build container cpu_limit = "500m" memory_limit = "1024Mi" # service containers service_cpu_limit = "100m" service_memory_limit = "200Mi" # helper container helper_cpu_limit = "100m" helper_memory_limit = "200Mi"Create a GitLab runner instance. Use the ConfigMap and secret created above in the runner instance:
apiVersion: apps.gitlab.com/v1beta2 kind: Runner metadata: name: example-runner namespace: podman-runner spec: config: podman-gitlab-runner-custom-toml gitlabUrl: 'https://gitlab.com' imagePullPolicy: Always tags: 'podman-runner' token: gitlab-runner-secret- Validate that the GitLab operator (runner controller manager) and
example-runnerpods are running in the namespace. This ensures that all above steps are followed correctly. - Make sure that you have a runner associated with the GitLab project. See Figure 2 indicating that a runner is successfully registered in the GitLab project. Notice the tag used is
podman-runner.
Create a Containerfile to validate the image build process using GitLab Pipeline. The example uses a Containerfile to install few required packages and copy source code and few additional commands. The goal is to showcase that we can build a container image with Podman GitLab runner in the same way as it can be done locally or a virtual machine:
FROM explore0.jfrog.io/base-repo/redhat.io/ubi8/python-38:1-94 LABEL MAINTAINER developer_name WORKDIR /app COPY app.py requirements.txt /app/ RUN pip install --trusted-host pypi.python.org -r requirements.txt EXPOSE 9000 ENV NAME Hello from Dockerfile ENV BGCOLOR blue CMD ["python","app.py"]Create a
.gitlab-ci.ymlfile that uses runner tagpodman-runner. Below is an example YML file for reference.Note that in the
build_imagestage,KUBERNETES_SERVICE_ACCOUNT_OVERWRITE: podman-sa and KUBERNETES_POD_ANNOTATIONS_1: "openshift.io/required-scc=podman-scc"is required.Additionally,
/home/user/entrypoint.shneed to be executed in theimage_buildstage because GitLab runner overrides the entrypoint defined in the image:image: registry.gitlab.com/o1266/podman-runner/podman-runner-cmd:3.0.0 default: tags: - podman-runner stages: # List of stages for jobs, and their order of execution - build_image - other_stage build_image_job: variables: # Below 2 lines are needed to make sure the runner uses appropriate service # account and scc. KUBERNETES_SERVICE_ACCOUNT_OVERWRITE: podman-sa KUBERNETES_POD_ANNOTATIONS_1: "openshift.io/required-scc=podman-scc" stage: build_image script: - /home/user/entrypoint.sh - podman build -t testimage:1.0 -f Containerfile . other_demo_job: stage: other_stage script: - echo "This is not build stage"Run GitLab pipeline. It should run
build_imageandother_stagefrom.gitlab-ci.yml.Every stage in the GitLab pipeline spins up a new container. For the
build_imagestage, the container comes up with SCCpodman-sccand service accountpodman-sa. This stage will build the container image. However, theother_stagecontainer comes up withrestricted-v2SCC and default service account.Sample output:
Running with gitlab-runner 17.2.1 (9882d9c7) on example-runner-runner-55dcf49df4-w5rw7 Txbc_BqZq, system ID: r_sK6jnT16Erqv Preparing the "kubernetes" executor 00:00 "ServiceAccount" overwritten with "podman-sa" "PodAnnotations" "openshift.io/required-scc" overwritten with "podman-scc" Using Kubernetes namespace: podman-runner Using Kubernetes executor with image registry.gitlab.com/o1266/podman-runner/podman-runner-cmd:3.0.0 ... Using attach strategy to execute scripts... Preparing environment 00:08 Using FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 1h0m0s... Waiting for pod podman-runner/runner-txbcbqzq-project-58314782-concurrent-0-9fisjq7z to be running, status is Pending Waiting for pod podman-runner/runner-txbcbqzq-project-58314782-concurrent-0-9fisjq7z to be running, status is Pending ContainersNotReady: "containers with unready status: [build helper]" ContainersNotReady: "containers with unready status: [build helper]" Running on runner-txbcbqzq-project-58314782-concurrent-0-9fisjq7z via example-runner-runner-55dcf49df4-w5rw7... Getting source from Git repository 00:04 Fetching changes with git depth set to 20... Initialized empty Git repository in /builds/o1266/podman-runner/.git/ Created fresh repository. Checking out 1b45e11f as detached HEAD (ref is main)... Skipping Git submodules setup Executing "step_script" stage of the job script 06:19 $ /home/user/entrypoint.sh $ podman build -t testimage:1.0 -f Containerfile . STEP 1/9: FROM registry.access.redhat.com/ubi9/python-312:latest Trying to pull registry.access.redhat.com/ubi9/python-312:latest... Getting image source signatures Checking if image destination supports signatures Copying blob sha256:95a3c7ae41d820ce47bbbff3848e46c36e6402431600de4457bfdf091901a772 Copying blob sha256:db22e630b1c7cf081461536c489254a8d1b39ceda32f8f3025314f032860d984 Copying blob sha256:cc296d75b61273dcb0db7527435a4c3bd03f7723d89a94d446d3d52849970460 Copying blob sha256:9f2fbf79f82326e925352def0dbdccc800cb9da2fe0125c4d1c33a9cbfc1b629 Copying config sha256:ca17740c85b9615346d509e099062b761a29692040b143af0689dc4f5bd11005 Writing manifest to image destination Storing signatures STEP 2/9: LABEL MAINTAINER developer_name --> 9c1e5c6f60fd STEP 3/9: WORKDIR /app --> e4e28c3cca48 STEP 4/9: COPY app.py requirements.txt /app/ --> 69fbbe770a66 STEP 5/9: RUN pip install --trusted-host pypi.python.org -r requirements.txt Collecting Flask (from -r requirements.txt (line 1)) Obtaining dependency information for Flask from https://files.pythonhosted.org/packages/61/80/ffe1da13ad9300f87c93af113edd0638c75138c42a0994becfacac078c06/flask-3.0.3-py3-none-any.whl.metadata Downloading flask-3.0.3-py3-none-any.whl.metadata (3.2 kB) Collecting Redis (from -r requirements.txt (line 2)) Obtaining dependency information for Redis from ........ ....... ....... --> 5960727b5065 STEP 6/9: EXPOSE 9000 --> ce4704e8ccc5 STEP 7/9: ENV NAME Hello from Dockerfile --> f1a2ba32abd7 STEP 8/9: ENV BGCOLOR blue --> 1e5d528d242d STEP 9/9: CMD ["python","app.py"] COMMIT testimage:1.0 --> 81a670dd615e Successfully tagged localhost/testimage:1.0 81a670dd615e5266143cb19d271c1685cc435b05e0af62511575603f500ab60a Cleaning up project directory and file based variables 00:00 Job succeeded
Note
The configurations are tested only on OpenShift 4.12.
Conclusion
Running Podman in rootless mode, as demonstrated in this article, offers numerous advantages. It eliminates the need for standalone build machines for runners or agents, enhances cost efficiency, enables ephemeral runners, and optimizes the use of the OpenShift platform for image building while leveraging native Kubernetes features.
While the article primarily focuses on setting up a GitLab runner, the initial steps for enabling Podman containers can also be applied to other CI tools like Jenkins, GitHub, and more. It also opens up possibilities for other integrations and automations.
Limitations
Nested containers are not supported up to OpenShift 4.15. OpenShift 4.16+ may have support for fully nested containers and users can execute commands such as podman run, podman compose, and etc.
Credits: The configurations related to Podman are provided by Charro Gruver. Charro's contribution significantly helped with this implementation.
