The Issue

Every team that outgrows a pile of Docker Compose files eventually asks The Question: do we need Kubernetes? And every team that answers “yes” then discovers the second question hiding behind it - how do we build with Kubernetes instead of around it?

Our Take

The first two posts in this series covered the ground floor: SOCKS serves your files as a single container, and TIBIA puts your workloads securely on the internet behind a second container. That covers a huge amount of real-world scope for teams serving up to several thousand customers, but sooner or later requirements start showing up that smell an awful lot like a workload scheduler, a cross-region control plane, or an infrastructure resource API - and the usual path from there is a cliff. Your existing infrastructure grew by evolution instead of design, so rather than squeezing yet another node into whatever gaps are left in your CIDR ranges, you decide to stand up a new cluster of fresh interchangeable nodes where any workload can land anywhere - because that’s the beauty of Kubernetes, right? Then to get it online you bolt on a second cloud load balancer, open a second set of firewall holes, and to get certificates working you bring a second certificate manager (with its own stores) to the party, too.

Now you have two front doors, two certificate architectures, and a doubled security surface for one domain that’s physically split across two infrastructure pools - with nothing in common but a DNS entry. Then 20% of your admin API calls start failing and your cluster’s customer facing workload latency p99 spikes through the roof - you’re left manually rebuilding nodes one at a time at 3:00 a.m. while your SLA continues to degrade as the cluster tries to shift your workloads to whatever compute is still standing. That’s a story we don’t like, let alone telling it more than once.

We wanted a better Operator experience - and we wanted to feel like we were working with Kubernetes, not like the cluster was dictating how we delivered services - no tail wagging the dog, here. Today we’re announcing our answer to the question: the Kubernetes Applied Responsibility & Placement Administration Layer, or KARPAL. It’s Open Source, it’s on our GitHub, it’s up-to-date, and it does one job: roll out a split-role K3s cluster where every workload placement is intentional, by convention - but not forced.

What is KARPAL?

KARPAL is a single-host K3s cluster built out of individual containers, where each one has a single narrow, predictable responsibility. Three servers run the control plane Raft (embedded etcd, HA), one ingress node runs the cluster’s internal Traefik instance and nothing else, and one worker node runs your application workloads. Each role scales independently - add servers for control-plane resilience, ingress nodes for edge capacity, workers for application capacity - without touching the other two. Ready to turn KARPAL into a multi-node cross-region Kubernetes cluster for physically segmented HA? Bring the node up, point the new KARPAL containers at any other node in the cluster, and… you’re done - time for a cup of coffee.

And no surprise - the whole thing sits behind our Open Source TIBIA domain gateway, which means we provide a direct upgrade path, and it also natively supports almost any existing Traefik (or pretty much any other cloud-native) gateway that you’re using. The domain gateway keeps owning Let’s Encrypt certificates, HTTP-to-HTTPS redirects, and the public and private entrypoints; KARPAL just adds one new Docker network named local-k3s and a cluster inside of it. Existing Docker services and new Kubernetes workloads then arrive through the same door - and since the outer layer owns all of TLS, nothing inside the cluster ever needs to deal with certificates.

Why We Built It

Here’s the dirty secret of most Kubernetes rollouts: the cluster doesn’t just run your workloads, it quietly duplicates everything you already had. A second load balancer. A second certificate manager with its own stores and its own renewal failures. A second (and third, and fourth) routing layer with its own opinions about (second, and third, and fourth) hostnames. A second set of firewall rules to keep in sync with the first. None of it is new capability - it’s the same job, done twice, by silo’d tooling and infrastructure pools that don’t know anything about each other. And just like most of what we do every day - every single copy is n+1 things to explain to the next person who inherits the setup.

We didn’t want to build a better copy - we wanted to delete the copies:

• One front door. Public traffic enters through the domain gateway you already trust, and the cluster sits behind it. There’s no second edge to expose, patch, or firewall.
• One certificate store. The gateway owns TLS so the cluster never needs to issue, renew, or store certificates. BYO, but only when you are ready.
• One routing story. A plain Docker container label, or a cluster Ingress, IngressRoute, or HTTPRoute - all resolve and route the way you expect.
• One firewall. Only the gateway’s ports are public, so your answer to “what’s exposed?” fits in a single sentence.

Our integrated split-responsibility model with manual cluster workload placement is what makes the deletion safe. With the control plane, the ingress, and your applications each pinned to their own node class, you always know where something runs and what else shares its fate - and a failure in one tier doesn’t take the other two with it. Instead of focusing on the best way to move to Kubernetes, we focused on the best way for Kubernetes to move in - and the measure of success is how little of the surrounding architecture has to change to make room for it. All of your Kubernetes workloads and existing Docker Compose services now share a front door, a certificate store, and a firewall - and suddenly your domain strategy is back to being something you can reason about (and respond to) without breaking a sweat.

What’s In The Box

• Three Role, Five Node Cluster: Three K3s servers, one ingress node, one worker node, each pinned to a static address range so a new node of any class has an obvious place to go.
• Every Ingress Type, Side By Side: Standard Kubernetes Ingress, Traefik IngressRoute CRDs, and the Gateway API (Gateway / HTTPRoute, standard-channel CRDs) are all enabled at once. Use whichever one your applications and tooling already speak - mix them freely.
• Separate Public and Private Traffic Lanes: Each traffic lane gets its own outer entrypoint, its own backhaul port into the cluster, and its own inner listener. A workload is published to a lane by intentionally attaching to it, never by accident - a route bound to the private lane is unreachable through public ports regardless of request shape.
• Verify Tooling: ./gateway verify and ./cluster helpers carry TIBIA’s bring-up and PASS/FAIL discipline into the cluster, and two whoami canaries (one Docker, one Kubernetes) give you all the copy/paste you need for getting your own workloads online immediately.
• Host Bring-Up Scripts: Installers for Docker, kubectl, and Helm, plus ufw and swap configuration, for Debian and Ubuntu on both x86-64 and arm64.
• Workload Templates: Every Kubernetes workload follows the same four-file shape (namespace, deployment, service, route), so adding your own app is copy, rename, and install.

Staging First, Production Second, Cluster Third

They say “third time is the charm” and we couldn’t agree more - KARPAL deliberately inherits TIBIA’s cautious bring-up rather than reinventing it. You start the gateway in BRINGUP MODE with staging certificates, verify it, then switch to PRODUCTION MODE and verify again - and only then do you bring up the cluster:

./cluster init
./cluster up

./cluster up flat-out refuses to run until init has completed, and the cluster comes up behind an edge that’s already proven to work. When something does go wrong, you know it’s the new part - and the README’s troubleshooting table maps the usual symptoms (a 404 from the outer edge versus a 404 from the inner Traefik, nodes stuck NotReady, a staging certificate stubbornly hanging around on prd) to the puzzle piece that’s actually at fault. What you’re left with is a production-grade Kubernetes stack ready for whatever tooling you use and workloads you want to run - KARPAL is a completely agnostic building block, not another framework for your team to adapt. ChatGPT - Can you help me get this working?

Honest Limitations

We’re dedicated to talking about the soft underbelly of our creations, and a cluster that lives inside Docker has a few worth knowing up front:

• Every K3s node runs as a privileged container. A container escape from a pod is a host compromise. That’s the cost of running K3s inside Docker; role-based placement limits what runs where, not what a node can do to the host. ChatGPT - What are the risks of doing this?
• The outer Traefik still mounts the Docker socket, so compromising it compromises the machine. A read-only socket proxy is on you if that tradeoff doesn’t suit you.
• The private lane is not authenticated - it’s separate, not secret. The K3s API and both dashboards are served through it, and who can reach it is decided entirely by your cloud firewall or security group. Docker can punch straight through ufw; verify connectivity configurations from an external machine.
• It’s only one host. Five containers means HA etcd and role separation, but not host-level redundancy. If the machine goes, everything goes - run multiple KARPAL clusters in a geographically segmented architecture for cross-regional HA.
• Placement is opt-in. The servers carry no taint, so a pod with no selector can still land on them unless you taint them yourself.
• Everything on the Docker Compose shared network (local-containers) can talk to everything else on it, and the cluster state (tokens, certificates, secrets, etcd data) lives in directories on disk that need to be backed up and kept out of git.
• Doing it - K3S_VERSION and TRAEFIK_VERSION are yours to move. Upgrades are rolling and one node at a time, and the release notes are required reading.

Get It / What’s Next

KARPAL is available, now, on our GitHub. You’ll need a Debian or Ubuntu host with at least 2vCPU and 2GB of RAM, a domain with its two DNS records pointed at the host, and public ports 80 and 443 reachable. The cluster half of the story, once your TIBIA gateway is verified in production mode, looks like this:

cd stack
./cluster init && ./cluster up
eval "$(./cluster kubeconfig)"
./cluster nodes
../workloads/k3s/traefik/install
../workloads/k3s/whoami/install

The README and its companion docs cover the full architecture, networking and certificate handling, firewall guidance, and how to deploy your own workloads in either lane. If you already run TIBIA, the changes you need are limited to one new network - see Built on TIBIA. As with the rest of the family, the code is out in the open so you can use it, break it, and improve it. If you hit a rough edge, find a gap in ./gateway verify, or want KARPAL to do something it doesn’t yet - open an Issue or send a Pull Request. The best fixes tend to come from someone else’s “oh, that’s why nobody does it that way” moment.

---

Disclosure: The “ChatGPT” links in this post are provided purely as a convenience, and 315Concepts does not endorse, review, or guarantee anything an AI assistant returns from them. AI-generated answers can be incomplete, outdated, or simply wrong - especially on security topics. Always use AI-provided information safely: treat it as a starting point, and double-check it directly against authoritative source material (such as the official Traefik and Docker documentation, and the project README) before acting on it, particularly for anything touching your firewall, certificates, or production infrastructure.