Implementing and updating tools in a Kubernetes cluster is a complex process, requiring careful preparation and strict adherence to procedures. This article presents the key steps and recommendations for implementing and updating two important components in a Kubernetes cluster: Portworx and prometheus-stack. The first part of the article discusses updating MetalLB and ArgoCD.

Before beginning the implementation of Portworx in the Kubernetes cluster with the use of ArgoCD, there are a couple of initial steps that are worth following through.

In order to efficiently manage Portworx with the help of ArgoCD, it is best to create a separate application dedicated to that service. The file structure should comply with the path declared in the ArgoCD application configuration.

When organizing the structures of  Portworx files, it is important to check that the path is consistent with the one declared in the ArgoCD application configuration. In the example above, a portworx folder was created, with the kustomization.yaml folder and the structure for eventual overlayers. The portworx.yaml file will contain a specific configuration for Portworx.

If Portworx has not been under the management of ArgoCD yet, it is essential to create the appropriate environment. As with MetalLB, this step will allow us to create a restore point. In the situation where an earlier configuration exists, it is advised to make sure to restore the environment.

In the next step, start the synchronization process of the application in ArgoCD, to implement the changes defined in the configuration files.

If the implementation process and the potential initial update have been properly executed, the Storage Cluster manifest should be now available for downloading on the Portworx panel. Below are the necessary steps for the realization of the manifest, as well as for the eventual update of the service.

In the portworx panel, in the section “Install and Run”, you should click on the “Download” option that appears when clicking on “Actions”.

After downloading the manifest, it is important to make sure that the entry  for portworx.io/install-source is the same as for key user= and c=, and then if the entry name: is the same as for portworx.io/install-source c=

If differences appear, it is advised to download data directly from the cluster and to insert it directly into the manifest. The ready manifest should be stored in the following file: /portwrox/overlays/dev/storage-cluster.yml.

When updating the Portworx service, it is enough to make changes in the appropriate sections of the manifest.

To change the Kubernetes version (kbver=):

Verification of entry (name:):

Verification of the (user=) section:

After implementing those changes, the Portworx service manifest is ready to be applied in the Kubernetes cluster. Do not forget to verify the correctness of the configuration and that the applied changes are consistent with the updating processes.

In the portworx panel, go to the “Install and Run” section and keep on clicking on “Next” until you reach the final step. In the “Portworx Operator” section, click on the eye icon next to the operator manifest, copy the Portworx Operator manifest, go to the /portworx/base/catalog, and create the portworx-operator.vml file. Then you just need to insert the copy of the Portwork Operator manifest into the new file.

After executing those steps, the Portworx Operator manifest will be available in the local repository, ready to be used in the configuration of the Kubernetes cluster. Check the correctness of the copied manifest and apply it following the requirements of the application.

To access the StorageClass manifest for Portworx, run the following command in the terminal to receive the YAML of the StorageClass manifest.

After this operation, the manifest will be displayed in the console. You can also save the manifest directly in the file with this command:

This step will save the StorageClass manifest to the portworx-storageclass.yml file.

After executing those steps, the StorageClass manifest for Portworx will become available, which you can then edit or use in other configuration processes. It is essential to monitor and verify the values in the manifest before its application.

To update the manifest through the Portworx panel, login to the platform and choose the current specification in the “Install and Run”. Then clone it, defining new versions for Portworx and the cluster.

It is essential to define a new version, for both Portworx and the cluster itself. In the next steps, it is required to adapt the specifications to our needs.

The final result is a familiar view from which you can download all the necessary manifests.

After the cloning process, download the manifest and insert it in the dedicated path for your environment f.e. /portworx/overlays/dev/. On the last resuming screen will be displayed information concerning the creation of a Secret type object.

In the situation where you have the right Secret object, there is no need to create another one. Thanks to this step, you can update the Portworx manifest, adapting it to the current trends.

In this step, it is also primordial to verify and potentially conduct some changes. These data are key, and if necessary, remember to replace them. Additionally, it is essential to create and add the appropriate Secret object that we previously mentioned.

To realize that, you can copy the Secret object directly from the cluster with the following command:

Then, add the copied data to the file in the dedicated path, f.e. /portworx/overlays/px-essential-secret.env. It is also worth remembering to not place the Secret in such a way in the version control system, like GIT. This is valid for all Secret objects, not just this particular one. If its addition to the repository is required, is it advised to encrypt it with a GPS key.

Before starting the update, it is required to set up the kustomization.yml files, which will connect all the previously created manifests. Each folder (/base/, /overlays/dev/, /overlays/prod) needs to have a /portworx/base/kustomization.yml file.

The folder /base/ nomen omen is used as a base. The manifests declared in the kustomization.yml file from this folder are the base for manifests from the /overlays/ folder.

/portworx/overlays/kustomization.yml

After executing those operations, you can compare the manifest with the cluster. This comparison can be made through ArgoCD after sending changes to the GIT repository. It is best to avoid the use of the Sync option and to prefer the Refresh one, as well as App Diff.

If ArgoCD shows no errors or changes that impact the functionality of the cluster, we can start the synchronization of the ArgoCD application. You can observe the changes on the cluster side with the following command:

After stabilizing the cluster, the update will be complete.

Updating the prometheus-stack into the cluster was one of the most time-consuming and demanding processes. It is mainly caused by the complexity of the Prometheus-stack itself which is made of several components and because of the importance of recreating ArgoCD from scratch.

Despite the many hours required to implement the Prometheus-stack with ARgoCD, it is a justified and necessary decision, especially in the context of long-term development of the cluster and future updates.

Before starting the update it is important to remember a couple of key steps – it is important to remember to verify that the created application in ArgoCD is called “monitoring” and that the definition of the application is located in the monitoring.yml file in the repository.

It is also important to verify that the structure of the files corresponds to the cluster’s requirements and that all the necessary files and configurations are available before starting the update. By taking care of those details at the preparation stage, complications are less likely to happen and it ensures a smooth updating process of the prometheus-stack into the cluster.

As a first step, it is necessary to prepare the script for the generation of the prometheus-stack with the help of the Helm tool.

Before the first generation of the manifest it is important to define the version that is currently used in the cluster. This way, you have a “clear” manifest for the current version. Because of the complexity and size of the prometheus-stack, the use of values.yml files can be very beneficial (and sometimes is required).

You will find below Helm’s instruction, to generate a manifest, taking into consideration the values from values.yml files and the ({{target-version}}):

Before the generation, it is worth getting acquainted with the documentation of the values.yml file. This file allows the adaptation of the manifest to the specifications of the cluster and other related services, by predefining parts of variables.

Thanks to this process of generating manifests has become more and more flexible and adapted to individual needs as well to the specifications of a given cluster.

In regards to the consequent size of the prometheus-stack manifest, an efficient approach is the local change comparison. The realization of this task by ArgoCD is therefore quite complicated because of the important quantity of data that has to be treated and showcased in the browser.

In the cluster’s situation, it is best to use different methods and use local change comparison. It is however best to create a restore point, to which we will refer when comparing the new manifests. This is valid for manual setup and for downloading the prometheus-stack manifest from the cluster.

This operation should be prepared from a strict manifesto for the current version of the cluster. It is worth using the previously prepared script or the command to generate templates. However, you should execute this command with the option –include-crds=false, which will drastically reduce the size of the file.

You can now proceed to the next steps of setting up the restore point and local change comparison in the manifests for the prometheus-stack.

Setting up the strict manifest for the current version available on the cluster can be made with the help of the previously prepared script or the command to generate templates. However, you should execute this command with the option –include-crds=false, which will drastically reduce the size of the file.

If the strict manifest is already available, it is worth creating a backup. To achieve this, create a file containing the same objects as the manifest. In other words, you need to create a copy of the raw manifest, by directly downloading the objects from the cluster.

The raw manifest can contain various objects, like f.e. Service, which will be displayed as follows:

The creation of a backup is made by copying the objects from the cluster and saving them in a file that will serve as a restore point in case of need to return to the original configuration.

It is advised that the download of the manifest object Service named monitoring-kube-prometheus-kube-controller-manager is made with the following command:

Marked sections can be removed successively. They will only interfere with the process of comparing files.

You should repeat this operation for each object from the raw manifest, according to their order. Each object from the manifest should be copied from the cluster and saved in the appropriate file, creating this way a restore point for specific configurations.

After copying all the objects, you need to proceed to a comprehensive file comparison. A practical operation that will facilitate your work is comparing the objects right after saving them in the prometheus-stack-old.yml file. This way, it is possible to make changes and corrections in the values.yaml file on an ongoing basis.

Similar to the situation with the previous application, this step is only necessary if prometheus-stack isn’t available from ArgoCD’s side. In such a situation, comparing the application to the current state of the cluster is essential.

Unfortunately, not all variables can be defined in the values.yaml file described earlier. Some settings do not find their equivalent there, which may require an alternative approach to define them.

In this situation, we can use the kustomize tool, which allows for the flexible addition of extra configurations, or even full manifests.

The kustomization.yml file is defined in the following way:

The patchesStrategicMerge mechanism allows you to flexibly overwrite only those variables or configurations that are of interest, leaving the rest intact. This keeps previous entries updated rather than completely replacing them.

Entry in the raw manifesto:

Content of the patch.yml file:

Result of kustomize build:

Additionally, it is essential to add a Secret object. This task should be performed the same way we did with “Portworx Secret”.

The updating process is pretty similar to the ones you went through for previous services – generate the manifest for the most recent version, taking into account the values.yml, patch.yml, and secrets.yml files.

The command for the generation of the manifest is presented as follows:

When generating the manifest for the new version, it is important to remember to change the switch – include-crds to true – as well as to separate the manifests, the same way we proceeded until now.

Then, the changes were compared using the App Diff tool in ArgoCD.  This allows to catch even the smallest changes that could have been missed earlier. After finalizing this process and making sure there are no key differences or errors, it is time for the synchronization of the application with ArgoCD.

If App Diff does not detect any major differences, you can safely continue and proceed to the synchronization of the application, guaranteeing consistency and compatibility of configuration with cluster expectations and requirements.

This was the second part of the post on upgrading the Kubernetes cluster to version 1.26. In this part, the upgrade processes for Portworx and Prometheus Stack services were discussed. In the next entry, we will cover the steps involved in upgrading Blackbox Exporter services and Kubernetes itself.