Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform

[aicam]

Problem

Today bin/k8s on main ships a single Helm chart that targets a local / on-premise Kubernetes cluster only. Our AWS (EKS) deployment runs the same Helm chart, but everything underneath it — the EKS cluster, S3 bucket, IAM/IRSA roles, Elastic IP, DNS, node pools, cert-manager — is created by hand and lives nowhere in the repo. None of that substrate is captured as code.

The result: standing up Texera on a fresh AWS account is an undocumented, manual prerequisite checklist that drifts and is hard to reproduce. We'd like to unify the local and AWS deployment stories so there's one documented, reproducible path from "empty cluster (or empty AWS account)" to "running Texera," while keeping the Helm chart itself cloud-agnostic.

Scope of "substrate"

The Helm chart already covers the application. The open question is only how we provision/document the substrate it runs on:

• EKS cluster + managed node pools
• IAM roles / IRSA for service accounts
• S3 bucket (LakeFS backend), Elastic IP, DNS, cert-manager / Let's Encrypt

Options considered

1. Helm only

Deploy the chart into a cluster that already exists. No AWS substrate creation.

Pros

• Simplest — one helm install
• Identical for local and AWS — fully cloud-agnostic
• Nothing extra to learn or maintain in the repo
• Keeps the Apache PR scoped to just the chart

Cons

• Doesn't create the cluster, S3 bucket, IAM, EIP, node pools, or cert-manager — all manual on a new AWS account
• New-AWS-account UX = a long prerequisite checklist done by hand
• Substrate stays undocumented-as-code → drift, hard to reproduce

2. eksctl

AWS CLI that provisions an EKS cluster + node pools + IAM/IRSA + add-ons from a small YAML, via CloudFormation. Helm still deploys on top.

Pros

• Purpose-built for EKS; minimal config for a working cluster
• Handles IRSA, node pools, and add-ons natively
• "New AWS account → cluster" in ~one command
• Config file is committable and reproducible

Cons

• AWS-only — no use for local or other clouds
• Doesn't manage non-cluster resources well (S3 bucket, EIP, DNS) — you still script those
• CloudFormation-based: slower, less granular state control
• Adds an AWS-specific tool to a repo that's otherwise portable

3. Terraform

Declarative IaC that can provision everything: EKS cluster, S3, IAM, EIP, DNS, node pools — and across clouds. Helm deploys on top (or via the Helm provider).

Pros

• Most reproducible & reviewable (full state, plan/apply, diffs)
• One tool for all substrate, not just the cluster
• Cloud-agnostic — same tool for AWS / GCP / local
• Can even invoke Helm itself, so one apply does substrate + deploy

Cons

• Heaviest to set up and maintain (modules, remote state, locking)
• Steeper learning curve; more moving parts
• State management is its own operational burden
• Overkill if you only ever target one EKS cluster

Our current position: eksctl

We're currently leaning toward eksctl, because:

• GCP and Azure are low priority for now — the cloud-agnostic advantage of Terraform doesn't pay off yet, so we'd rather not take on its maintenance cost speculatively.
• We prefer easier maintenance over easier deploying. eksctl keeps the AWS-specific config small and declarative while letting the Helm chart stay fully portable, rather than introducing Terraform modules + remote state + locking that we'd have to operate.
• eksctl handles the genuinely fiddly EKS-specific parts (IRSA, node pools, add-ons) natively, and the cluster config is committable.

Regardless of the choice, a small amount of non-cluster substrate (S3 bucket, EIP, DNS) would still be scripted alongside.

We'd love your input

Which approach should we standardize on for upstreaming AWS deployment into bin/k8s? Please vote with a reaction on this post and share your reasoning in the comments — especially if you deploy Texera (or similar) on a cloud we haven't prioritized:

• 👍 — Helm only
• 🚀 — eksctl (our current lean)
• ❤️ — Terraform

[chenlica]

Thanks for starting the discussion. We want to support both cloud-based and on-prem deployments. Please confirm if eksctl supports both.

@zuozhiw @shengquan-ni @Yicong-Huang @sshiv012 @parshimers Any input from you?

[sshiv012]

I'm still getting familiar with the AWS side of the deployment, but here's what stood out reading through bin/k8s.

@chenlica standard eksctl does not support on-prem. It's the AWS-only CLI for provisioning Amazon EKS clusters as @aicam mentioned. What should be common for both flavors of deployment is Helm. Not sure if we use EKS ANYWHERE or just barebones k8s on prem, if its eks anywhere then we can maybe reuse it.

One thing I wasn't sure was fully captured in the framing: it might help to separate the application chart from the substrate, because as far as I can tell only the substrate is really in question.
On the chart side, texera-helm already looks closer to cloud-agnostic than I expected, the persistence templates only create hostPath PVs when the storage class is local-path (minio-persistence.yaml, postgresql-persistence.yaml), so other classes fall through to dynamic provisioning, and LakeFS already targets blockstore.type: s3 rather than MinIO-specific APIs. If I'm reading that right, moving to EBS + real S3 is mostly a values change, not a template change.

That leaves the substrate i.e cluster, node pools, IRSA, S3, EIP, DNS, cert-manager as the only layer the eksctl-vs-Terraform question is actually about.

On eksctl: I agree it's the right tool for the cluster layer (node groups, Pod Identity). My one hesitation is that it doesn't get us all the way to "one path to reproduce the deployment" — as @aicam noted, S3/EIP/DNS are out of scpe for it, so if those stay manual we may partly recreate the undocumented-setup problem we're trying to solve. That's where Terraform's plan is appealing: it warns before it deletes a data bucket or changes IAM, detects drift if someone clicks around in the console, and the code doubles as documentation.

But I think it really comes down to who owns the deployment:

• Project maintainers (keep-it-simple, rotating contributors): eksctl + Helm + a committed bootstrap script is probably the lighter path.
• General adopters (integrating into a real org/projects): Terraform is the better thing to offer — portable, composable into existing IaC, self-documenting, and usually the tool they already speak.

So maybe it isn't either/or. We can consider (eventually) shipping a thin, optional Terraform module (or even just documented .tf for S3/IAM/IRSA plus a reference eksctl cluster config), so maintainers stay on the simple path while adopters get something that drops into their stack.

[parshimers]

eksctl is just for Amazon's distribution of k8s. Which is fine, as stated GCP/Azure are way less popular, and on prem is always unique. The important thing is that there is no hard binding to AWS and that's already the case.

I think having a well documented set of steps for setting up everything on AWS is a very good idea. IAM in particular is a nightmare without knowing exactly what each service actually needs. People running on prem will have their own particular setups that will need interpretation anyway. For example they probably have some object store, but each one will have it's own configurations for buckets and so on. It's not possible for us as developers to say how that's done on each system a priori. There should be guidelines (e.g. what addresses and ports the various services need, what object stores are known to work, etc.) but not specifics. Maybe an example with some particular set of services would be a good intermediate step, because then the administrator would be able to draw analogies to their environment.

I have gone through Milvus's setup on AWS and that was about as straightforward as it could be. It really would have taken me a very long time to figure out things on my own without that walkthrough.

[aicam]

Thanks everyone for your help and suggestions, based on the discussion, we decided to go with Helm + eksctl without Terraform. The main reason is that the people who deploy the service are technical people and we can let them handle the detail they need and we provide base requirements for them to deploy.

[chenlica]

@aicam Please confirm why we don't want to use Terraform.

[aicam]

The main reason we do not use Terraform is that we currently focus on AWS and onprem only.

[chenlica]

@aicam To make the reason clear, Terraform doesn't support AWS nor on-prem, right?

[sshiv012]

Terraform supports AWS, Azure, GCP and other leading cloud providers. As Ian pointed out for on-prem is highly specific, but terraform should support application/platform stuff like helm release, Postgres, MinIO, LakeFS, and the cluster layer for VMs. It would also be self documenting in nature.

The only other complexity I see with it and isn't discussed is for terraform, we would need to manage, store and track the state of deployments, for a team this would be in some datastore that's collaborative like S3, for local deployments or smaller scale or single admin it can be stored locally.

[chenlica]

I still don't see a clear reason why Terraform is not considered. @aicam Please be more specific about the reasoning.

[chenlica]

@aicam : To make it more clear, please answer the following questions:

• Does Teraform supports both AWS and on-prem?
• If so, what is "overkill"?

[aicam]

@chenlica Terraform indeed support AWS and on-prem, it sits on top of our stack (Helm) and manage infrastructure. It is used to define set of services and resources to deploy Helm charts on specific cloud providers including Heroku, AWS, GCP, Azure, ... . Our current plan is to standardize our Helm charts and use eksctl to deploy the application on AWS and on-prem.
Terraform is overkill because it is not required for on-perm deployment (including people who use their own clusters) and is also not mandatory for AWS because eksctl is the native AWS tool. In the future when we want to add GCP or any other cloud, we will most likely replace eksctl with Terraform but the efforts to standardize and clean our Helm charts is orthogonal to Terraform or eksctl so we prioritize cleaning our Helm charts and later once we have them, we can add Terraform on top of that.

[chenlica]

Thanks for the clear explanation.

[zuozhiw]

I second @aicam here, I also use both terraform and helm charts for work. As Ali said, they are at different layers, and using terraform for one or two envs is a bit overkill and its value start to show more clearly after we need to manage more of envs, terraform has this benefit of managing infra as code, but it also brings a bit extra complexity.

[chenlica]

@zuozhiw Thanks for the confirmation. Now we have a clear decision.

[aicam]

Thanks @madhusudancs, thanks for your comments today, this is the discussion we had previously

[chenlica]

@madhusudancs, it will be great to have more advice from you. Thank you.

[aicam]

To update our discussion, we decided to go with Terraform after more discussions with industry experts.