Avoid tight coupling to external dependencies in your Helm charts, as they may break your upgradability, directly impacting your users. In this article, Miguel De Barros, Principal Architect Payment Solutions, deep dives into the challenges our team has faced using Helm for Kubernetes-based Mojaloop releases, the impact it has had on users, and what we’re changing in Mojaloop v14 to simplify future deployments.
One of the most important objectives of Mojaloop as an open source product is that it should be easily deployable and runnable on almost any kind of underlying infrastructure platform. There are two issues here: the “easy to deploy” aspect and the matter of “running on almost any kind of underlying infrastructure platform.” Mojaloop has overcome the infrastructure platform issue by leveraging Kubernetes as our target platform, which allows us to simplify and abstract our underlying infrastructure in combination with IaC (Infrastructure as Code) to automate the challenging components. This has allowed us to resolve the “easy to deploy” aspect by utilising Helm, which will be my focus here. We will discuss the most important lessons learned in using Helm, especially when it comes to external dependencies, and how we overcame them.
What is Helm, and why do we use it?
Helm is a package manager that helps you deploy, view, and upgrade software applications that target a Kubernetes runtime. If we distill its intention down, it is very similar to Linux package management systems like apk, yum, or dpkg tools that you may be familiar with from Linux-based operating system. Similarly, Helm is used to streamline the management of Kubernetes-based applications, allowing deployers to focus more on the configuration of Mojaloop (i.e. how do I want it to run?) and less on the underlying technical details (i.e. what container version of X component do I need to complete the Y feature?).
Ultimately, Helm allows us to share a “packaged” fully-tested version of Mojaloop. The package contains “charts” with a collection of templatized Kubernetes descriptors, configurations, and potentially both internal and external dependencies. Take note of the external dependency there because it could hurt you in the long run – more on that in a moment.
Our experience using Helm and lessons learned
We have wrestled with Helm throughout its progression from v1 to v3 over the last two years of developing the Mojaloop platform. Here are the most challenging issues that we’ve come across during this effort:
- New versions of Kubernetes often introduce breaking changes or deprecate certain functionality. As a result, one needs to continuously maintain one’s Helm charts to ensure support for the latest Kubernetes deployments.
- External Helm dependencies often introduce breaking changes, or in some cases are even deprecated. This is to be expected as external charts are managed and maintained by third parties.
Unfortunately, the first point is unavoidable as an integral part of maintaining a Helm chart. The good news is that the changes made to the Mojaloop Helm charts to support the newer Kubernetes versions have rarely impacted the upgradability of a Mojaloop deployment. This means that for the most part, Mojaloop users can upgrade their deployments seamlessly as they transition their Kubernetes platform to later versions. We’ll discuss a further edge-case here that will follow on from the second point above.
The latter issue has negatively impacted Mojaloop users to a greater degree. On several occasions, we were forced to publish major versions that were not backward compatible as a knock-off effect from the changes or deprecations made to the external dependency that were outside of our control. Thusly, Mojaloop users could not easily upgrade their existing Mojaloop deployments. Instead, they had to follow a migration path as discussed in the official Mojaloop Upgrade Strategy Guide – a non-trivial process. In essence, the “upgradability” of your Helm chart is directly linked to the external dependencies that you choose!
Remember that “edge-case” mentioned above? Well, let’s consider the following scenario:
- Your Helm charts support Kubernetes v1.21 with an external dependency.
- You want to support Kubernetes v1.22 which introduced a breaking change with the Ingres API.
- You make the necessary changes to your charts to support the v1.22 changes.
- You attempt to deploy your chart but fail to deploy even though you made your changes.
- You discover that your external dependencies were not updated to support the Kubernetes v1.22 API changes. In turn, your Helm charts do not support v1.22 due to the tight coupling with these dependencies! 🙈
Once again, we find that your external dependencies dictate the maintainability and upgradability of your chart.
Consider the Mojaloop v13 release (see figure A.). We have an external database (such as MySQL) configured as an external dependency in this scenario. The benefit of this approach is that with a single step (i.e., command), you will be able to install all of Mojaloop’s components, and it supports backend-end dependencies (i.e., MySQL, etc.). This method is great for the “easy to deploy” objective from above. However, there is a cost, and that cost will directly impact your upgradability due to the tight-coupling of dependencies.
How we are resolving the external dependency issue
With the introduction of v14, our proposal to remove all external dependencies from the Mojaloop Helm Charts was accepted by the Mojaloop Design Authority. Instead, we will manage these external dependencies through configuration.
We have also integrated a common Helm utility and helper function library to assist with this approach, providing the following primary benefits:
- Standard helper functions to configure back-end dependencies consistently, which is important to our configuration-based approach.
- Support templatized configuration values, which will reduce the need for hard-coded values, make it easier to configure a Mojaloop installation going forward, and simplify our installation – especially for development environments (see below for more information).
How does this impact the “easy to deploy” objective?
Instead of a single step, a user will have to follow this approach (see figure C.):
- Install all external dependencies (e.g. MySQL, Kafka, etc.)
- Create a custom configuration for the Mojaloop installation against the external dependencies
- Install Mojaloop with the customised configuration
This approach adds more steps. However, it resolves the tight coupling of external dependencies and ensures best practices when deploying Helm charts in general (i.e., external dependencies should always be managed separately).
To further streamline this process, we have included an “example-backend” wrapper chart that does deploy all external dependencies. It is important to note that we do not recommend using this for a production-grade installation, but you can use it for development or PoC purposes.
We are able to further simplify the deployment process, at least for development or PoC environments, by using the “example-backend” wrapper chart for the following steps (see figure B.):
- Install external dependencies using the “example-backend” wrapper chart
- Install Mojaloop with the default configuration
Wrapping it up
We end where we started: Avoid tight coupling to external dependencies in your Helm charts, as they may break your upgradability, directly impacting your users.
This will enable you to offer better support to your users by providing a more frictionless upgrade experience, ensuring that your Helm charts are maintainable, and demonstrating that you are in complete control of your “Helm” (charts).
Feel free to reach out directly through the comment section below or contact our team if you would like to learn more or contribute to the Mojaloop Open Source Project and help us make the v14 release a reality!