Update Openshift Container Platform 4.x to minor and major version in restricted network environment

Main goals of updates/upgrades — bug fixes, new features, security vulnerabilities fixes, where ideal state is always be up to date. Updates/upgrades can be done to the latest minor version of the existing current major version or to the next major version. For instance, if you deployed your cluster when the latest version was 4.3.22 you can upgrade it today to the latest minor 4.3.40 version or gradually to the latest existing version, i.e OCP 4.6.7. In restricted networks this process includes additional steps of mirroring the relevant images in your existing private registry and changing configuration of several cluster components that will allow you to perform upgrade smoothly.

This document based on the next official Red Hat documentation links

Updating a restricted network cluster

Updating a cluster between minor versions

OpenShift Container Platform (OCP) 4 upgrade paths

Backing up ETCD Data

Configuring the Samples Operator

1. Define your cluster upgrade path

To upgrade your specific cluster to the latest minor version you don’t need to perform upgrade path check, so you can jump directly to the next chapter. But if you want to upgrade to the major version you need to know what is your cluster version upgrade path. To define it, perform the next:

# export CURRENT_VERSION=4.3.22
# export CHANNEL_NAME=stable-4.5
# curl -sH 'Accept:application/json' "https://api.openshift.com/api/upgrades_info/v1/graph?channel=${CHANNEL_NAME}" | jq -r --arg CURRENT_VERSION "${CURRENT_VERSION}" '. as $graph | $graph.nodes | map(.version=='\"$CURRENT_VERSION\"') | index(true) as $orig | $graph.edges | map(select(.[0] == $orig)[1]) | map($graph.nodes[.].version) | sort_by(.)'
OUTPUT
[]

In the field I was able to upgrade from latest minor version to the latest minor version of the next Major version, for example, from OCP 4.3.22 → 4.3.40(latest for today minor of 4.3 version) → OCP 4.4.30 → OCP 4.5.20 → OCP 4.6.7 and bypass upgrade path check.

Warning: please don’t try to jump directly from major version 4.3 to 4.5, for instance, since this imaginative shortcut will only cost your time and frustration. Avoid that!

If you anyway want to check your upgrade path, it means you can’t upgrade directly from existing version, 4.3.22, to any version in 4.4 or 4.5 major version, so next action is to define to what version inside 4.3 you can upgrade. For that, change your CHANNEL_NAME environment variable accordingly.

# export CHANNEL_NAME=stable-4.3
# curl -sH 'Accept:application/json' "https://api.openshift.com/api/upgrades_info/v1/graph?channel=${CHANNEL_NAME}" | jq -r --arg CURRENT_VERSION "${CURRENT_VERSION}" '. as $graph | $graph.nodes | map(.version=='\"$CURRENT_VERSION\"') | index(true) as $orig | $graph.edges | map(select(.[0] == $orig)[1]) | map($graph.nodes[.].version) | sort_by(.)'
OUTPUT
[
  "4.3.40"
]

Now you know that your next version for upgrade should be 4.3.40. And only after you will complete to upgrade to this latest minor version, you will repeat the steps above with CHANNEL_NAME=stable-4.4 to understand how to continue with upgrade path.

2. Mirroring the OpenShift Container Platform image repository

This step assume that you have external mirror registry and internal mirror registry ready with existing version repositories (It was required for your cluster deployment). On your external mirror registry server (one that have connection to the Internet)
Set the required environment variables:

# export OCP_RELEASE=4.3.40
# export LOCAL_REGISTRY=
'registry.ocp43.prod.sales.lab.tlv.redhat.com:5000'
# export LOCAL_REPOSITORY='ocp4.3.40/openshift4.3.40'
# export PRODUCT_REPO='openshift-release-dev'
# export ARCHITECTURE=x86_64
# mkdir /opt/registry/ocp4.3.40
# export REMOVABLE_MEDIA_PATH=/opt/registry/ocp4.3.40
# cd /opt/registry/ (this is my base registry folder)
Check the content of the json file you prepared in deployment process that including your mirror registry.
# For example, our one called pull-secret2.json
OUTPUT
cat pull-secret2.json | jq
{
  "auths": {
    "cloud.openshift.com": {
      "auth": "<your pull secret>",
      "email": "<your email address>"
    },
    "quay.io": {
      "auth": "<your pull secret>",
      "email": "<your email address>"
    },
    "registry.connect.redhat.com": {
      "auth": "<your pull secret>",
      "email": "<your email address>"
    },
    "registry.redhat.io": {
      "auth": "<your pull secret>",
      "email": "<your email address>"
    },
    "registry.ocp43-prod.sales.lab.tlv.redhat.com:5000": {
      "auth": "<your base64-encoded credentials>"
    }
  }
}
# export LOCAL_SECRET_JSON="/opt/registry/pull-secret2.json"
# export RELEASE_NAME="ocp-release"
# podman login quay.io 
OUTPUT
Authenticating with existing credentials...
Existing credentials are valid. Already logged in to quay.io
Review the images and configuration manifests to mirror:
# oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=${REMOVABLE_MEDIA_PATH}/mirror quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} --dry-run
OUTPUT
info: Mirroring 109 images to registry.ocp43-prod.sales.lab.tlv.redhat.com:5000/ocp4.3.40/openshift4.3.40 ...
Mirror the images and configuration manifests to a directory on the removable media:
# oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=${REMOVABLE_MEDIA_PATH}/mirror quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE}
OUTPUT
info: Mirroring 109 images to registry.ocp43-prod.sales.lab.tlv.redhat.com:5000/ocp4.3.40/openshift4.3.40 ...
Next very significant step is to prepare your images to be moved to the restricted network.

After completing those steps you can tar you relevant REMOVABLE_MEDIA_PATH folder content and move with this tar file to whitening process and after that proceed the next steps in the Internal Registry.

Take the media to the restricted network environment and upload the images to the local container registry.
# oc image mirror --from-dir=${REMOVABLE_MEDIA_PATH}/mirror 'file://openshift/release:${OCP_RELEASE}*' ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} --registry-config=${LOCAL_SECRET_JSON}
To check that you have all relevant images in the repository you are just uploaded to your registry run the next:
# curl https://registry.ocp43-prod.sales.lab.tlv.redhat.com:5000/v2/ocp4.3.40/openshift4.3.40/tags/list -k -u admin:*******(password)
OUTPUT
{"name":"ocp4.3.40/openshift4.3.40","tags":["4.3.40-kube-etcd-signer-server","4.3.40-cli-artifacts","4.3.40-oauth-proxy","4.3.40-cluster-kube-controller-manager-operator","4.3.40-service-ca-operator","4.3.40-baremetal-operator","4.3.40-cluster-csi-snapshot-controller-operator","4.3.40-cluster-svcat-apiserver-operator",.........................................................."4.3.40-local-storage-static-provisioner","4.3.40-etcd"]}
One of the files in the mirror folder is very significant for the next step in upgrade process, this is image_signature_file, for example, mirror/config/signature-sha256-81154f5c03294534.yaml. You need to copy this file to your installer VM that have access to your cluster.

3. Create ConfigMap, Upgrade the Cluster and Edit your cluster ImageContentSourcePolicy

Next steps will be performed from installer machine of your cluster

To create the config map please run the next command:
# oc apply -f <image_signature_file>
For <image_signature_file>, specify the path and name of the file, for example, mirror/config/signature-sha256-81154f5c03294534.yaml that you copied from the internal registry server to your installer machine
To upgrade the restricted network Cluster
Install oc version 4.4.8 or later
# export LOCAL_REGISTRY=
'registry.ocp43.prod.sales.lab.tlv.redhat.com:5000'
# export LOCAL_REPOSITORY='ocp4.3.40/openshift4.3.40'
# oc adm upgrade --allow-explicit-upgrade --to-image ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}<sha256_sum_value>
The <sha256_sum_value> value is the sha256 sum value for the release from the image signature ConfigMap, for example, @sha256:81154f5c03294534e1eaf0319bef7a601134f891689ccede5d705ef659aa8c92
To edit your Cluster ImageContentSourcePolicy
# oc get ImageContentSourcePolicy
# oc edit ImageContentSourcePolicy image-policy-0
Change ocp4.3.22/openshift4.3.22 to ocp4.3.40/openshift4.3.40 and save
# oc edit ImageContentSourcePolicy image-policy-1
Change ocp4.3.22/openshift4.3.22 to ocp4.3.40/openshift4.3.40 and save

Now you can go to your Ocp console → Administration → Cluster settings → Details and check. This process will run in background and can take a while. At the end of the process your cluster will be up to date (With latest minor version of the same major version) At some point when you will check your update status you might see the next notification: “Unable to apply 4.3.40: the cluster operator openshift-samples is degraded” And by running oc get co you can see that openshift-samples cluster operator really in degraded state

# oc get co | grep openshift-samples
openshift-samples   4.3.40    True        True          True       38h

To fix this issue you need to add your registry to configs.samples.operator.openshift.io/cluster resource, since if samplesRegistry not defined, update of sample images trying to go to redhat.io to pull the relevant images and it can’t do it in restricted network environments.

# oc edit configs.samples.operator.openshift.io/cluster
spec:
  architectures:
  - x86_64
  managementState: Managed
  samplesRegistry: registry.ocp43-prod.sales.lab.tlv.redhat.com:5000

Now by running oc get co you can see that openshift-samples cluster operator in Available state

# oc get co | grep openshift-samples
openshift-samples   4.4.13    True        False         False      38h

At the end of the process you can check your current version by running

# oc version
Client Version: 4.6.7
Server Version: 4.3.40
Kubernetes Version: v1.17.0+3288478

5. Upgrade your cluster to major version

Now, after successful update to the latest minor version, you can continue and upgrade to the major version. Before you will continue, please repeat steps in paragraph 2 (mirror your registry with required version images, for instance 4.4.30).

And now repeat steps in paragraph 3 and change accordingly all values to the latest version in 4.4, in our case it was 4.4.30.
Now you can go to your Ocp console → Administration → Cluster settings → Details →Channel and change to Stable-4.4

This process will run in background and can take a while. At the end of the process your cluster will be up to date (With latest minor version of the major version). You can encounter the same issues with openshift-samples operator during this upgrade — solution explained above.

Repeat the same steps in upgrade toward 4.5.20 — the latest 4.5 version and then toward 4.6.7 — the latest and greatest long term support version.

Enjoy your updated clusters!