The Helm Team is proud to announce the first stable release of Helm 4. New Features Helm 4 has numerous new features, but a few deserve highlighting here: Redesigned plugin system that supports We...
The Helm Team is proud to announce the first stable release of Helm 4.
Helm 4 has numerous new features, but a few deserve highlighting here:
For full release notes, please see: https://helm.sh/docs/overview/
Helm v4 is a major version with backward incompatible changes including to the flags and output of the Helm CLI and to the SDK.
Please evaluate the changes to your workflows. The changes are not as extensive as those from Helm v2 to v3, with the goal that the majority of workflows remain compatible between Helm v3 and v4.
Helm charts apiVersion v2 (majority of today's charts) will continue to be supported in Helm v4. Existing charts should continue to install, upgrade, and otherwise work. Please test the installation and upgrade of charts to ensure it works as expected. Changes (e.g., server side apply) may impact the experience.
The community keeps growing, and we'd love to see you there!
Download Helm v4.0.0. The common platform binaries are here:
The Quickstart Guide will get you going from there. For upgrade instructions or detailed installation notes, check the install guide. You can also use a script to install on any system with bash.
The Helm project has enjoyed code contributions from many community members. Many more community members have assisted by filing issues and working with us to identify and eliminate bugs while adding new features. The #helm-users slack channel has long been a friendly and open forum for getting help and learning more about Helm. We cannot thank you enough for making this a helpful, friendly, and welcoming community for all.
❤️ The Helm Team
After some work with kubernetes, i must really say, helm is a complexity hell. I'm sure it has much features but many aren't needed but increase the complexity nonetheless.
Also, please fix the "default" helm chart template, it's a nightmare of options and values no beginner understands. Make it basic and simple.
Nowadays i would very much prefer to just use terraform for kubernetes deployments, especially if you use terraform anyway!
Helm is my example of where DevOps lost it's way. The insanity of multiple tiers on templating an invisible char scoped language... it blows my mind that so many of us just deal with it
Nowadays I'm using CUE in front of TF & k8s, in part because I have workloads that need a bit of both and share config. I emit tf.json and Yaml as needed from a single source of truth
The problem with Kubernetes, Docker and anything CNCF related is what happens when everyone and their dog tries to make a business out of an OS capability with venture capital.
shudders.. `| nindent 12`..
I've been trying to apply CUE to my work, but the tooling just isn't there for much of what I need yet. It also seems really short-sighted that it is implemented in Go which is notoriously bad for embedding.
> seems really short-sighted that it is implemented in Go
CUE was a fork of the Go compiler (Marcel was on the Go team at the time and wanted to reuse much of the infra within the codebase)
Also, so much of the k8s ecosystem is in Go that it was a natural choice.
> CUE was a fork of the Go compiler (Marcel was on the Go team at the time and wanted to reuse much of the infra within the codebase)
Ah, that makes sense, I guess. I also get the feeling that the language itself is still under very active development, so until 1.0 is released I don't think it matters too much what it's implemented in.
> Also, so much of the k8s ecosystem is in Go that it was a natural choice.
That might turn out to be a costly decision, imho. I wanted to use CUE to manage a repository of schema definitions, and from these I wanted to generate other formats, such as JSON schemas, with constraints hopefully taken from the high-level CUE.
I figured I'd try and hack something together, but it was a complete non-starter since I don't work within the Go ecosystem.
Projects like the cue language live and breathe from an active community with related tooling, so the decision still really boggles my mind.
I'll stay optimistic and hope that once it reaches 1.0, someone will write an implementation that is easily embedded for my use-cases. I won't hold my breath though, since the scope is getting quite big.
Why don't you work with the Go ecosystem? You don't use K8s, terraform, etc? What ecosystem do you prefer?
what language would you have chosen?
> I wanted to use CUE to manage a repository of schema definitions, and from these I wanted to generate other formats, such as JSON schemas, with constraints hopefully taken from the high-level CUE.
Have you tried a Makefile to run cue? There should be no need to write code to do this
We evaluated CUE, Jsonnet and CDK8s when we wanted to move on from Helm, and ended up using CDK8s. It's proven to be a good pick so far, it's in Typescript.
Back when my job involved using Kubernetes and Helm, the solution I found was to use `| toJson` instead: it generates one line that happens to be valid YAML as well.
Both Jsonnet and CUE are implemented in Go which happens to be the language Helm is written in. While I agree that it reduces "general embedability" it's ripe fruit for Helm to integrate either or both of these as alternatives to YAML templating.
Holos[1] is an interesting project I’ve been looking at trying out.
I've looked at Holos recently
1. it seems like development has largely ceased since Sept
2. it looks to only handle helm, not terraform, I'm looking for something to unify both and deal with dependencies between charts (another thing helm is terrible at)
cue and argocd here. it is pretty neat.
the tf is still in hcl form for now.
RIP Ksonnet, we hardly knew what we were missing
jsonnet is the main DX issue therein
I don't think I've ever seen a Helm template that didn't invoke nightmares. Probably the biggest reason I moved away from Kubernetes in the first place.
We have several Helm charts we've written at my job and they are very pleasant to use. They are just normal k8s templates with a couple of values parameterized, and they work great. The ones people put out for public consumption are very complex, but it isn't like Helm charts have to be that complex.
In my book the main problem with Helm charts is that every customization option needs to be implemented by the chart that way. There is no way for chart consumer to change anything the chart author did not allow to be changed. That leads to these overly complex and config heavy charts people publish - just to make sure everything is customizable for consumers.
I'd love something that works more like Kustomize but with other benefits of Helm charts (packaging, distribution via OCI, more straight forward value interpolation than overlays and patches, ...). So far none have ticked all my boxes.
Yeah too many times the Helm chart is barely less complex than writing all the manifests yourself because all the manifest options are still in the chart
fluxCD brings a really nice helm-controller that will allow to change manifests via a postRenderers stub while still allowing to use regular helm tooling against the cluster.
https://fluxcd.io/flux/components/helm/helmreleases/#post-re...
Yeah, but then it is yet another layer of configuration slapped on top of the previous layer of configuration. That can't be the best solution, can it? Same thing for piping helm template through Kustomize.
Yeah, this setup is both nice and insane. If you don't need much extra customization it's great. But I have a setup where I needed both postBuild and postRenderer's + actual kustomization layering and it was awful trying to figure out the order of execution to get the right final output.
In hindsight it would have been much faster to write the resources myself.
Use helm to generate the manifests with a Makefile, use Kustomize to change said manifests for prod, staging, etc.
Kustomize can render Helm charts. It's "very basic" as in Kustomize will call the Helm binary to render the template, ingest it and apply patches.
I wrote a tool called "easykubenix" that works in a similar way, render the chart in a derivation, convert the YAML to JSON, import JSON into the Nix module structure and now you're free to override, remove or add anything you want :)
It's still very CLI deploy centric using kluctl as the deployment engine, but there's nothing preventing dumping the generated JSON (or YAML) manifests into a GitOps loop.
It doesn't make the public charts you consume any less horrible, but you don't have to care as much about them at least
Yes, this is the key. Helm charts should basically be manifests with some light customization.
Helm is not good enough to develop abstractions with. So go the opposite way: keep it stupid simple.
Pairing helm with Kustomize can help a lot as well. You do most of the templating in the helm chart but you have an escape hatch if you need more patches.
That's generally what I try to push for in my company.
A single purpose chart for your project is generally a lot easier to grok and consume vs what can be done.
I think the likes of "kustomize" is probably a more sane route to go down. But our entire infrastructure is already helm so hard to switch that all out.
I've personally boiled down the Helm vs. Kubernetes to the following:
Does your Kubernetes configuration need to be installed by a stranger? Use Helm.
Does your Kubernetes configuration need to be installed by you and your organization alone? Use Kustomize.
It makes sense for Grafana to provide a Helm chart for Grafana Alloy that the employees of Random Corp can install on their servers. It doesn't make sense for my employer to make a Helm chart out of our SaaS application just so that we can have different prod/staging settings.
This has been my argument for years now.
I think it is because most engineers learn to use Kubernetes by spinning up a cluster and then deploying a couple of helm charts. It makes it feel like that’s the natural way without understanding the pain and complexity of having to create and maintain those charts.
Then there are centralised ‘platform’ teams which use helm to try and enforce their own templating onto everything even small simple micro services. Functionally it works and can scale, so the centralised team can justify their existence but as a pattern it costs everyone a little bit of sanity.
I'm ashamed to say it but I cannot for the life of me understand how kustomize works. I could not ever figure out how to do things outside the "hello world" tutorials they walk you through. I'm not a stupid person (citation needed lol), but trying to understand the kustomize docs made me feel incredibly stupid. That's why we didn't go with that instead of Helm.
Helm requires you to write a template and you need to know (or guess) up front which values you want to be configurable. Then you set sane defaults for those values. If you find a user needs to change something else you have to edit the chart to add it.
With Kustomize, on the other hand, you just write the default as perfectly normal K8s manifests in YAML. You don't have to know or care what your users are going to do with it.
Then you write a `kustomizatiom.yaml` that references those manifests somehow (could be in the same folder or you can use a URL). Kustomize simply concatenates everything together as its default behaviour. Run `kubectl kustomize` in the directory with `kustomization.yaml` to see the output. You can run `kubectl apply -k` to apply to your cluster (and `kubectl delete -k` to delete it all).
From there you just add what you need to `kustomization.yaml`. You can do a few basics easily like setting the namespace for it all, adding labels to everything and changing the image ref. Keep running `kubectl kustomize` to see how it's changing things. You can use configmap and secret generators to easily generate these with hashed names and it will make sure all references match the generated name. Then you have the all powerful YAML or JSON editing commands which allow you to selectively edit the manifests if you need to. Start small and add things when you need them. Keep running `kubectl kustomize` at every step until you get it.
this, our helm charts are flat and for year only passed in the image as variable
What did you move to?
Infrastructure as code should from the beginning have been through a strict typed language with solid dependency and packaging contract.
I know that there are solutions like CDK and SST that attempt this, but because the underlying mechanisms are not native to those solutions, it's simply not enough, and the resulting interfaces are still way too brittle and complex.
Yup that's what SST wraps (or at least it did when I was fiddling with it). And even Pulumi still is at the behest of the cloud providers... it still has to mirror complexity of the providers to a considerable degree. Devexp is leaps and bounds better with Pulumi than CDK though.
I mean terraform provides this but using it doesn't give a whole lot of value, at least IME. I enforce types but often an upstream provider implementation will break that convention. It's rarely the fault of the IAC itself and usually the fault of the upstream service when things get annoying.
I don't think I want to use kubernetes (or anything that uses it) again. Nightmare of broken glass. Back in the day Docker Compose gave me 95% of what I wanted and the complexity was basically one file with few surprises.
If you can confidently get it done with docker-compose, you shouldn't even think about using k8s IMO. Completely different scales.
K8s isn't for running containers, it's for implementing complex distributed systems: tenancy/isolation and dynamic scaling and no-downtime service models.
One of the problems seems to be that most moderately complex companies where any one system would be fine with Compose would want to unify their operations, thus going to a complex distributed system with k8s. And then either your unified IT/DevOps team is responsible for supporting all systems on k8s, or all individual dev teams have to be competent with k8s. Worst case, both.
no-downtime is table stakes in 2025. I can't look at anyone in the eyes and tell them that our product is going to go down for a bit everytime we deploy (it'd also be atrocious friction for frequent deployment).
You can do no-downtime deploy of a web service with:
- Kamal
- Docker compose with Caddy (lb_try_duration to hold requests while the HTTP container restarts)
- Systemd using socket activation (same as Docker compose, it holds HTTP connections while the HTTP service restarts)
So you don't have to buy the whole pig and butcher it to eat bacon.
> - Systemd using socket activation (same as Docker compose, it holds HTTP connections while the HTTP service restarts)
Nit: it holds the TCP connections while the HTTP service restarts. Any HTTP-level stuff would need to be restarted by the client. But that’s true of every “zero downtime” system I’m aware of.
Being successful enough that any amount of downtime is an existential risk is a great problem to have. 99.99% don't have that problem; even huge successful businesses can survive unplanned downtimes (see: recent major outages).
It's far from table stakes and you can absolutely overengineer your product into the ground by chasing it.
"0 downtime" system << antifragile systems with low MttR.
Something can always break even if your system is "perfect". Utilities, local disasters, cloud dependencies.
Docker Compose still takes you 95% of what you need. I wish Docker Swarm survived.
What happened to it?
I'm still using it with not a single issue (except when is messes up the iptables rules)
I still confidently, upgrade the docker across all the nodes, workers and managers and it just works. Not a single time that it caused an issue.
Docker the company bet big on Swarm being the de facto container orchestration platform for businesses. It just got completely overshadowed by k8s. Swarm continues to exist and be actively developed, but it’s doomed to fade into obscurity.
For some reason I assumed it was unsupported. That doesn't seem to be the case.
> I wish Docker Swarm survived.
I heard good things about Nomad (albeit from before Hashicorp changed their licenses): https://developer.hashicorp.com/nomad
I got the impression it was like a smaller, more opinionated k8s. Like a mix between Docker Swarm and k8s.
It's rare that I see it mentioned though, so I'm not sure how big the community is.
I’d wager that like half the teams (at least) using kubernetes today should be using Nomad instead. Like the team I’m on now where I’m literally the only one familiar with Kubernetes and everyone else only has familiarity with more classic EC2-based patterns. Getting someone to even know what Helm does is its own uphill battle. Nomad is a lot more simple. That’s what I like about it a lot.
For better or for worse its a orchestrator (for containers/scripts/jars/baremetal) full stop.
Everything else is composable from the rest of the hashicorp stack consul(service mesh and discovery),vault(secrets) allowing you to use as much/or as little as you need and truly able to scale to a large deployment as needed.
In the plus column , picking up its config/admin is intuitive in a way that helm/k8s never really comes across.
Philosophy wise can put it int the unix way of doing things - it does one thing well and gets out of your way , and you add to it as you need/want. Whereas k8s/heml etc have one way or the high way - leaving you fighting the deployment half the time.
Mitchel Hashimoto was a genius when it came to opinionated design and that was Hashicorp's biggest strength when it was part of their culture.
It's a shame Nomad couldn't overcome the K8s hype-wagon, but either way IBM is destroying everything good about Hashicorp's products and I would proceed with extreme caution deploying any of their stuff net-new right now...
> I wish Docker Swarm survived.
Using it in prod and also for my personal homelab needs - works pretty well!
At the scale you see over here (load typically served on single digit instances and pretty much never needing autoscaling), you really don't need Kubernetes unless you have operational benefits from it. The whole country having less than 2 million people also helps quite a bit.
I only whish terraform was more recognized by upstream projects, like postgres, tailscale, ingress operators.
A one-time adoption from kubectl yaml or helm to terraform is doable - but syncing upstream updates is a chore.
If terraform (or another rich format) was popular as source of truth - then perhaps helm and kubectl yaml could be built from a terraform definition, with benefits like variable documentation, validation etc.
I've embraced kustomize and I like it. It's simple enough and powerful enough for my needs. A bit verbose to type out all the manifests, but I can live with it.
This is what I've done too. Just enough features easily available to handle everything i've ever needed in the simple deployments I use. Secrets, A/B configuration, even "dynamic reload" of a Deployment for Configmap changes.
Gets the job done.
I'm using sed on my yaml files. Currently considering kustomize instead, but I wouldn't touch Helm with a 10 foot pole.
Incidentally, Terraform is the only way I want to use Helm at all. Although the Terraform provider for Helm is quite cumbersome to use when you need to set values.
Kustomize with ArgoCD is my go to
Do you have any resources regarding using tf to handle deployments ?
I’d love to dig a bit.
Just use https://registry.terraform.io/providers/hashicorp/kubernetes... instead of helm...
…but how do you install helm charts via terraform?
Is there a helm provider?
If not, what would be the right way to install messy stuff like nginx ingress, cert-manager, etc.?
There is a helm provider. Why would you need it? Can't you just use the kubernetes provider?
People probably don't realize, that helm mostly is templating for the YAMLs kubernetes wants (plus a lot of other stuff that increases complexity).
There are many applications which are distributed as helm charts. Those charts install multiple deployments, service accounts and whatnot. They barely document all these things.
So if you want to avoid helm, you gotta do a whole lot of reverse-engineering. You gotta render a chart, explore all the manifests, explore all the configuration options, find out if they're needed or not.
An alternative is to just use helm, invoking it and forgetting about it. You can't blame people for going the easy way, I guess...
Yep, this 100%. Every time there is a technology which has became the "de facto" standard, and there are people proposing "simpler alternatives", this is the kind of practical detail that makes a GIANT difference and that's usually never mentioned.
Network effect is a thing, Helm is the de facto "package manger" for Kubernetes program distribution. But this time there are generally no alternative instructions like
tar xzf package.tar.gz; ./configure; make; adduser -u foo; chown -R foo /opt/fooI think we are having different contexts here. I am mostly talking about selfwritten services and how writing, maintaining and deploying helm charts for them is a nightmare.
Regarding dependencies: Using some SaaS Kubernetes (Google GKE) for example, you'll typically use terraform for SQL and other Services anyway (atleast we do use Google CloudSQL and not some selfhosted postgres in k8s).
I find it interesting that cert-manager points to kubectl for new users and not helm: https://cert-manager.io/docs/installation/
But, for sure, there may be reasons to use helm, as you said. I'm sure it is overused, though.
It might feels natural to try and use terraform to deploy kubernetes resources since you’ve likely configured the cluster using it, but the helm/kubeneters/kubectl providers are limited by terraform’s way of working. So whilst the providers try to marry the two when deploying anything complex it generally ends up feels like a hack and you lose a lot of the benefits of using terraform in the first place.
In my experience, it’s best to bootstrap ArgoCD/flux, rbac and cloud permissions those services need in Terraform and then move on to do everything else can via Kustomize via gitop. This keeps everything sane and relatively easy to debug on the fly, using the right tool for the job.
I know at least a couple times just the templating side saved me where it was convenient to just run a helm command with --dry-run to get the yaml and grab & modify the relevant pieces and apply those manually where I don't necessarily want the whole package or I want snippets of a package or modified yaml that their helm chart didn't support out of the box, etc.,.
Have you seen ingress-nginx.yaml? It's like 1000 lines with only a few cloud-provider specific changes. Managing and updating that manually with pure kubernetes sucks.
The kubernetes provider mostly just works exactly as you expect
Could you explain this a bit? Is helm optional part of the k8s stack?
Yes, you really don't need to use helm if you have terraform. Just use https://registry.terraform.io/providers/hashicorp/kubernetes... .
If you used helm + terraform before, you'll have no problem understanding the terraform kubernetes provider (as opposed to the helm provider).
It does make it challenging to track operators as upstream usually only provide/document helm installation.
If you write your own tf definition of operator x v1, it can be tricky to upgrade to v2 - as you need to figure out what changes are needed in your tf config to go from v1 to v2.
Helm is not official or blessed or anything, just another third party tool people install after install k8s.
The way I understand, helm is the npm of k8s.
You can install, update, and remove an app in your k8s cluster using helm.
And you release a new version of your app to a helm repository.
The thing i would add to this is that in most cases, you need to manually provide config values to the install.
This sounds okay in principle, but I far too often end up needing to look through the template files (what helm deploys) to understand what a config option actually does since documentation is hit or miss.
Helm is sort of like a docker (or maybe docker compose) for k8s, in terms of a helm chart is a prepackaged k8s "application" that you can ship to your cluster. It got very popular very quickly because of the ease of use, and I think that was premature which affects its day-to-day usability.
It's a client-side preprocessor essentially. The K8s cluster knows nothing about Helm as it just receives perfectly normal YAMLs generated by Helm on the client.
I really appreciate the k3s default with HelmChart type and operator installed. Makes working with charts simpler in my view
Yes, I use flux which has a similar HelmChart/HelmRelease resource. One of the things that took me a while to "get" with K8s is operators are just clients running on the cluster.
[dead]
Helm is truly a fractal of design pain. Even the description as a "package manager" is a verifiable lie - it's a config management tool at best.
Any tool that encourages templating on top of YAML, in a way that prevents the use of tools like yamllint on them, is a bad tool. Ansible learned this lesson much earlier and changed syntax of playbooks so that their YAML passes lint.
Additionally, K8s core developers don't like it and keep inventing things like Kustomize and similar that have better designs.
Imho, anyone who thought putting 'templating language' and 'significant whitespace' together is a good idea deserves to be in the Hague
Seriously. I’ve lost at least 100 hours of my life debugging whitespace in templated yaml. I shudder to think about the total engineering time wasted since yaml’s invention.
You blame YAML but I blame helm. I can build a dict in Python and dump it as YAML. I've painlessly templated many k8s resources like this. Why can't we build helm charts in a DSL or more sensible syntax and then dump k8s manifests as YAML? Using Go templating to build YAML is idiocy and the root of the issue here.
There's lots of advice on StackOverflow against building your own JSON strings instead of using a library. But helm wants us to build our own YAML with Go templating. Make it make sense.
I 100% agree. It’s not so much the yaml as it is the templating. I originally wanted to say “since the invention of yaml/jinja” in the parent comment because that’s what I’ve gotten most of my gray hairs from (saltstack templating). Go templates are not jinja but fundamentally the same thing - they have no syntax awareness and effectively are just string formatters.
I took out the part about templating because I thought it made my comment too wordy, but ended up oversimplifying.
This is more or less the approach Apple's "pkl" takes.
You define your data in the "pkl language", then it outputs it as yaml, json, xml, apple property list, or other formats.
You feed in something like:
apiVersion = "apps/v1"
kind = "Deployment"
metadata {
name = "my-deployment"
labels {
["app.kubernetes.io/name"] = "my-deployment"
["app.kubernetes.io/instance"] = "prod"
}
}
spec {
replicas = 3
template {
containers {
new {
name = "nginx"
}
new {
name = "backend"
}
}
}
}
And then you `pkl eval myfile.pkl -f yaml` and get back: apiVersion: apps/v1
kind: Deployment
metadata:
name: my-deployment
labels:
app.kubernetes.io/name: my-deployment
app.kubernetes.io/instance: prod
spec:
replicas: 3
template:
containers:
- name: nginx
- name: backend
The language supports templating (structurally, not textually), reuse/inheritance, typed properties with validation, and a bunch of other fun stuff.They also have built in package management, and have a generated package that provides resources for simplifying/validating most kubernetes objects and generating manifests.
There's even a relatively easy path to converting existing YAML/JSON into pkl. Or the option to read an external YAML file and include it/pull values from it/etc (as data, not as text) within your pkl so you don't need to rebuild everything from the ground up day 1.
Aaaaand there's bindings for a bunch of languages so you can read pkl directly as the config for your app if you want rather than doing a round trip through YAML.
Aaaaand there's a full LSP available. Or a vscode extension. Or a neovim extension. Or an intellij extension.
The documentation leaves a bit to be desired, and the user base seems to be fairly small so examples are not the easiest to come by... but as far as I've used it so far it's a pretty huge improvement over helm.
Yaml wouldn't be so bad if they made the templates and editors indent-aware.
Which is a thing with some Python IDEs, but it's maddening to work on anything that can't do this.
autocmd FileType yaml setlocal et ts=2 ai sw=2 nu sts=0
I'm sure Emacs and others have something similarwe use cue straight to k8s resources. it made life way better.
but we don't have tons of infra so no idea how it would run for big thousands-of-employees corps.
Helm sucks.
Helm, and a lot of devops tooling, is fundamentally broken.
The core problem is that it is a templating language and not a fully functional programming language, or at least a DSL.
This leads us to the mess we are in today. Here is a fun experiment: Go open 10 helm charts, and compare the differences between them. You will find they have the same copy-paste bullshit everywhere.
Helm simply does not provide powerful enough tools to develop proper abstractions. This leads to massive sprawl when defining our infrastructure. This leads to the DevOps nightmare we have all found ourselves in.
I have developed complex systems in Pulumi and other CDKs: 99% of the text just GOES AWAY and everything is way more legible.
You are not going to create a robust solution with a weak templating language. You are just going to create more and more sprawl.
Maybe the answer is a CDK that outputs helm charts.
Ok, thought experiment: why not use the k8s JSON interfaces and use jq to generate/template your deployments/services/statefulsets/argo images/etc.?
You say you want a functional DSL? Well, jq is a functional DSL!