The Operator Framework was paradigm shifting for managing infrastructure and providing services to users on Kubernetes by introducing a methodology for capturing a human operator's knowledge as a Kubernetes-native application, an Operator. The Operator Lifecycle Manager (OLM) brought systematic installation, removal, and update management for Operators. In parallel, the rise and recognition of the power and need for Git-based operations (GitOps) has brought much needed consistent, repeatable, and scalable management of not only Kubernetes and its surrounding ecosystem, but the management of the entire platform as a software product with full software development lifecycle (SLDC) for all the reasons end user applications have been refining their SDLC for all these years.
Challenge
OLM only offers two modes for operator installation and upgrade management: Manual and Automatic. In Automatic mode, the OLM will automatically update to the latest version of the given Operator as it becomes available in a subscribed channel. In Manual mode, the user must manually update the target minimal Operator version, including manually approving the installation and manually incrementing through required, phased updates. This prevents a user from moving between versions directly (ex: 1.1 to 1.5) because the user needs to move incrementally up to the target version (1.2, 1.3, and 1.4), resulting in a laborious process.
In the case of Automatic mode, you can easily capture in Git the required OLM Kubernetes custom resources in order to set the target Operator for automatic updates in a given channel. Oftentimes, an Operator maintains a latest channel as well as a channel for each major X version in semantic version (semver) format with no breaking changes in minor Y versions.
In the case of Manual mode, while you can set the startingCSV in the Subscription, you then have to manually approve the InstallPlan once it becomes available. While the Subscription can be captured in git, since the InstallPlan is automatically generated by the OLM, additional intervention is required given that the InstallPlan is not tracked. One potential option is to capture the generated InstallPlan in Git, set the approved value to true and have the resource be reconciled. However, these additional steps of manual intervention goes against the spirit of Git-based operations.
This ultimately leaves us with two options:
- Use Git-based operations and set Operators to use the automatic update approval strategy and hope that an update doesn’t come through that doesn’t put you in a situation of untested automatic braking production updates.
- We all recently experienced the (when, not if) eventually of this globally in late July 2024, when security software vulnerability definitions loaded at the kernel level in Microsoft Windows by a third-party tool were automatically updated with a corrupted vulnerabilities definition file, which caused infinite boot loop errors bringing down production systems and disrupting global air travel, banking, and government services among others
- Break the flow of Git-based operations and manually control Operator versions, but benefit in avoiding unexpected production changes.
Or….
Using the operators-installer Helm chart
Luckily, in our open source software based world, if you do not like any of your options, you can create a new one.
Enter option 3, the community built and maintained operators-installer Helm chart used in production by a community of individuals who also encountered these challenges in their production operations.
This Helm chart solves the challenge of GitOps for OLM managed Operators. It accomplishes this by providing a Helm chart you can invoke directly, use as a dependent Helm chart, or as a Helm chart called by kustomize (my preferred method) which allows you to set the approval mode to Manual and specify an operator version, and then the Helm chart will take care of automatically approving the appropriate InstallPlan(s) for you, which the Helm chart would conveniently assume is what you wanted if you were incrementing the Operator version.
Additionally, the chart also handles incrementally upgrading operator versions which avoids the time consuming process of manually determining and then stepping through each required intermediate version.
And lastly, if there are any Operators that you want to place in automatic approval mode, that can be accomplished as well, which allows for a consistent way to manage operators regardless of approval method.
The result with option 3 is that you can truly achieve declarative Git-based operations with fixed Operator versions with intuitive controlled Operator updates!
Examples
The following are several examples demonstrating how the Helm chart can be used. Additional methods can also be found within the chart documentation and tests.
Helm called by kustomize
Install the external-secrets-operator using operators-installer via kustomize:
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
helmCharts:
- name: operators-installer
repo: https://redhat-cop.github.io/helm-charts
version: 3.0.1
valuesInLine:
operatorGroups:
- name: external-secrets-operator
createNamespace: true
targetOwnNamespace: false
operators:
- channel: stable
installPlanApproval: Manual
name: external-secrets-operator
source: community-operators
sourceNamespace: olm
csv: external-secrets-operator.v0.8.1
namespace: external-secrets-operatorAs a Helm subchart
Instal the external-secrets-operator using operators-installer via Helm.
Chart.yaml:
---
apiVersion: v2
name: example-my-org-ops
description: example declarative install of operator
type: application
version: 1.0.0
dependencies:
- name: operators-installer
version: "3.0.1"
repository: "https://redhat-cop.github.io/helm-charts"operator-installtions-values.yaml:
---
operators-installer:
operatorGroups:
- name: external-secrets-operator-2
createNamespace: true
targetOwnNamespace: false
operators:
- channel: stable
installPlanApproval: Manual
name: external-secrets-operator
source: community-operators
sourceNamespace: olm
csv: external-secrets-operator.v0.8.1
namespace: external-secrets-operator-2
installPlanVerifierActiveDeadlineSeconds: 1200Direct from Helm
CLI:
helm upgrade -i --repo=https://redhat-cop.github.io/helm-charts operators-installer -f operator-installtions-values.yamlCaveats
- The operators-installer Helm chart is community maintained and not supported by Red Hat.
- While the operators-installer Helm chart can automatically increment through Operator versions, support is not currently available to verify whether an Operator upgrade causes issues other than verifying the installation of the new Operator version was successful. The end user should test and verify any operator upgrades in lower environments to ensure no issues occur, such as API changes.
- As the Operator Framework does not support the ability to perform rollbacks, the operators-installer Helm chart does not either.
Getting help
If you are excited about this solution to solve your current problems, reach out to the community for free (as in pizza) help by opening a Git issue.
Future
There are currently two features in Tech Preview status in Red Hat products that are intended to address the root challenge of declarative Git-based operations for upgrading operators managed by OLM.
OLM v1.0 is designed to solve many of these problems and as of Red Hat OpenShift 4.16 is in Tech Preview. This is done by deprecating the pre 1.0 OLM custom resource definitions and introducing ClusterExtension which, among other things, replaces the old Automatic and Manual modes and startingCSV in the Subscription with a version parameter which supports declaring ranges.
Additionally, Red Hat Advanced Cluster Manager has a new OperatorPolicy custom resource in Tech Preview that takes a similar approach to the operators-installer Helm chart which could be a great supported alternative if you are a user of Advanced Cluster Management, once it leaves Teach Preview status.
In the meantime, we have the operators-installer.
