Run Kubernetes with k3s
A real Kubernetes cluster on one VM, private until you say otherwise.
Shelley handles the setup. About 10 minutes.
Prefer a terminal?
curl -fsSL https://exe-staging.dev/deploy/k3s/prompt.txt | ssh exe.dev new --prompt=/dev/stdink3s is a CNCF-certified Kubernetes in one binary, with ingress (Traefik), a load balancer, local storage and CoreDNS built in. Here it runs under systemd on an exe.dev VM with a real disk: a cluster for learning, CI, Helm chart tests, or one copy per pull request.
What exe.dev adds: the Kubernetes API never faces the internet (you use
kubectl over SSH). Ingress routes get HTTPS and exe.dev login from the
HTTPS proxy, with no cert-manager or auth proxy, and apps
see the caller in X-ExeDev-Email. Pods reach the LLM gateway and other
integrations without API keys. Shelley, the agent on
the VM, can run kubectl for you.
What you get#
- k3s v1.37.2+k3s1 running as a systemd service, with pod and service networks that don't clash with the VM's own.
- kubectl and Helm working for
exedevwithout sudo. - Traefik answering on port 8000, with a sample app at the VM's private URL that shows your exe.dev email.
- PersistentVolumeClaims backed by the VM's disk through the local-path provisioner.
- Nightly backups of the cluster state, server token and volumes, with the first one checked.
What's running, and where
https://my-k3s.exe.xyz/
Traefik, serving your Ingress routes (a sample app to start)
https://my-k3s.exe.xyz:9898/
Any LoadBalancer Service on a port from 3000 to 9999
ssh -L 16443:localhost:6443 my-k3s.exe.xyz
The Kubernetes API, for kubectl on your computer, over SSH only
Deploy with Shelley#
Click Deploy and Shelley will set up k3s on a new VM, then check that it works. No config-file wrangling required.
Prefer a terminal?
curl -fsSL \
https://exe-staging.dev/deploy/k3s/prompt.txt \
| ssh exe.dev new --prompt=/dev/stdinWhen it's done, run this on your computer (only you can):
ssh my-k3s.exe.xyz cat /etc/rancher/k3s/k3s.yaml | sed 's/:6443/:16443/' > my-k3s.kubeconfig # then KUBECONFIG=my-k3s.kubeconfig
ssh -N -L 16443:localhost:6443 my-k3s.exe.xyzWhat Shelley is told
Set up k3s on this VM by following the exe.dev recipe at https://exe-staging.dev/deploy/k3s.md from top to bottom.
The recipe's example VM name is my-k3s. This VM's name is the output of `hostname`: use it wherever the recipe writes my-k3s. Skip the step that creates the VM; you are on it.
You are done when all of these are true. Check each one yourself:
- k3s v1.37.2+k3s1 running as a systemd service, with pod and service networks that don't clash with the VM's own.
- kubectl and Helm working for `exedev` without sudo.
- Traefik answering on port 8000, with a sample app at the VM's private URL that shows your exe.dev email.
- PersistentVolumeClaims backed by the VM's disk through the local-path provisioner.
- Nightly backups of the cluster state, server token and volumes, with the first one checked.
Read the whole recipe, including the closed details, and follow
"Install by hand" from top to bottom: the k3s config, the Traefik
config, k3s itself, Helm, the sample app, and the backup timer. Use the
pinned versions as written. The walkthrough sections are for the owner
afterwards: don't install podinfo or other charts, add nodes, set up
Tailscale, copy the VM, or run the Restore, Update or Uninstall
instructions during setup. Leave the VM private and do not reboot.
Check `kubectl get nodes` shows one Ready node named k3s, every pod in
kube-system is Running or Completed, and `curl localhost:8000` returns
the whoami output. Finish with the VM's private URL and the two
commands that give the owner kubectl on their own computer.
Finish with the URLs I should open and these commands, which only I can run, from my own computer (with this VM's name in them):
ssh my-k3s.exe.xyz cat /etc/rancher/k3s/k3s.yaml | sed 's/:6443/:16443/' > my-k3s.kubeconfig # then KUBECONFIG=my-k3s.kubeconfig
ssh -N -L 16443:localhost:6443 my-k3s.exe.xyz
Also at /deploy/k3s/prompt.txt. The recipe it follows is /deploy/k3s.md.
Try the cluster#
On your computer, ssh my-k3s.exe.xyz. Then, on the VM:
kubectl get nodes
kubectl get pods -A
Expect one node, k3s, Ready, and the system pods Running.
Open https://my-k3s.exe.xyz/. After exe.dev login you'll see the sample
app, whoami, echoing your request,
including X-Exedev-Email: you@example.com. Its manifest is
~/k3s/whoami.yaml; swap in your own app when ready.
Use kubectl from your computer#
Install kubectl 1.36-1.38. On your computer, fetch the kubeconfig, pointed at local port 16443:
mkdir -p ~/.kube
ssh my-k3s.exe.xyz cat /etc/rancher/k3s/k3s.yaml \
| sed 's/127.0.0.1:6443/127.0.0.1:16443/' > ~/.kube/my-k3s.yaml
chmod 600 ~/.kube/my-k3s.yaml
Open the tunnel in one terminal, use it from another:
ssh -N -L 16443:localhost:6443 my-k3s.exe.xyz
export KUBECONFIG=~/.kube/my-k3s.yaml
kubectl get nodes
Helm, k9s and Lens use the same KUBECONFIG. The file is cluster-admin:
guard it like an SSH key.
Put an app on the web#
Both routes are private (exe.dev login) until you share them.
Ingress, at https://my-k3s.exe.xyz/. Traefik listens on port 8000,
where the proxy sends the VM's main URL. Leave out host in your rules,
as the sample does, so they also match copies of the VM. Use path prefixes
for several apps.
LoadBalancer, at https://my-k3s.exe.xyz:<port>/. k3s's
ServiceLB
opens the Service's port on the VM, and the proxy forwards ports 3000 to
9999:
apiVersion: v1
kind: Service
metadata:
name: myapp
spec:
type: LoadBalancer
selector:
app: myapp
ports:
- port: 3000 # on the VM, and in the URL
targetPort: 8080 # in the pod
NodePorts (30000-32767) aren't proxied.
Share with ssh exe.dev share add my-k3s alice@example.com
(Sharing). Traefik passes X-ExeDev-Email and
X-ExeDev-UserID through, so apps can authorize by email
(Login with exe). Trust them only on requests from
the proxy: other pods can call your Service directly with any headers.
Go public with ssh exe.dev share set-public my-k3s. That opens port
8000, so every Ingress route; ports 3000-9999 stay private. Undo with
share set-private.
Keep data on the disk#
The default StorageClass, local-path,
stores each PersistentVolumeClaim as a directory under
/var/lib/rancher/k3s/storage, included in the nightly backup:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data
spec:
accessModes: [ReadWriteOnce]
storageClassName: local-path
resources:
requests:
storage: 5Gi
Sizes aren't enforced; volumes share the VM's disk. Deleting a claim deletes its data.
Install charts with Helm#
Try podinfo, whose chart ships its own tests. On the VM:
helm install podinfo oci://ghcr.io/stefanprodan/charts/podinfo \
--version 6.17.0 --namespace podinfo --create-namespace \
--set service.type=LoadBalancer --wait
helm test podinfo -n podinfo
Tests end in Phase: Succeeded; the app is at
https://my-k3s.exe.xyz:9898/. Remove it with
helm uninstall podinfo -n podinfo. For your own chart: helm lint,
helm install --wait, helm test, helm uninstall.
Run your own images#
No registry needed: build with Docker on the VM and import into k3s.
docker build -t myapp:v1 .
docker save myapp:v1 | sudo k3s ctr images import -
Reference it as docker.io/library/myapp:v1 with
imagePullPolicy: Never, and bump the tag on each build.
For a private registry, put credentials in
/etc/rancher/k3s/registries.yaml and sudo systemctl restart k3s. For
ghcr.io, with a read:packages-only token:
configs:
ghcr.io:
auth:
username: your-github-user
password: ghp_your_read_only_token
More in k3s's private registry docs. To keep that token off the VM, exe.dev's container registry integrations mint short-lived pull tokens (untested with k3s).
Call APIs without keys in Secrets#
Pods use the VM's network, so they reach its
integrations. Point OpenAI- or Anthropic-compatible
clients at https://llm.int.exe.xyz/v1 with no API key
(LLM gateway). For other APIs, an
HTTP proxy integration adds the
credential at the edge, so it's never in a Secret or a backup. See
agent secrets. Any pod can use them, so run only
workloads you trust.
A cluster per pull request#
A copy of the VM is the same cluster, apps and volumes included, at a new URL. On your computer:
ssh exe.dev cp my-k3s my-k3s-pr-42
About a minute later https://my-k3s-pr-42.exe.xyz/ is up. Deploy the
branch there (helm upgrade, kubectl set image), share the link, and
ssh exe.dev rm my-k3s-pr-42 after merging. This relies on the pinned
node name k3s; see Configure k3s.
For CI, keep one VM as a warm template and copy it per job; see GitHub Actions and CI, or ask Shelley to script it.
Add nodes#
Warning
Untested on exe.dev so far.
VMs share no private network and all have 10.42.0.42 on eth0, so run
flannel over Tailscale (the Tailscale integration
joins without an auth key on the VM). On the server, add to
/etc/rancher/k3s/config.yaml, then sudo systemctl restart k3s:
node-ip: 100.x.y.z # tailscale ip -4
node-external-ip: 100.x.y.z
flannel-iface: tailscale0
tls-san: [100.x.y.z]
On each agent, /etc/rancher/k3s/config.yaml:
server: https://100.x.y.z:6443 # the server's Tailscale address
token: K10... # sudo cat /var/lib/rancher/k3s/server/node-token on the server
node-ip: 100.a.b.c # this agent's Tailscale address
node-external-ip: 100.a.b.c
flannel-iface: tailscale0
Then curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.2+k3s1 sh -s - agent.
Local-path volumes stay on the node that made them. k3s also has an
experimental Tailscale integration.
Limits#
- One node, no control-plane failover.
- No cloud load balancer. ServiceLB opens ports on the VM; only HTTP on 8000 and 3000-9999 reaches the outside. Raw TCP/UDP needs SSH tunnels or Tailscale.
- No network block storage. Volumes are directories on one disk.
- TLS is the proxy's, so no cert-manager for exe.xyz names. For your own domain, see custom domains.
The nuts and bolts#
Shelley's setup steps, and your maintenance reference.
Install by hand
Create a VM#
On your computer (pick another name if my-k3s is taken):
ssh exe.dev new --name=my-k3s
ssh my-k3s.exe.xyz
Everything else runs on the VM, as exedev.
Configure k3s#
Three settings matter on exe.dev:
- Networks. The VM's own network is
10.42.0.0/16, k3s's default pod network. Move pods and services to kubeadm's default ranges. - Node name. Pinned to
k3s, so VM copies keep their pods and volumes. - Kubeconfig. Group-readable, so you and Shelley need no sudo.
sudo mkdir -p /etc/rancher/k3s
sudo tee /etc/rancher/k3s/config.yaml >/dev/null <<'EOF'
cluster-cidr: 10.244.0.0/16
service-cidr: 10.96.0.0/16
cluster-dns: 10.96.0.10
node-name: k3s
write-kubeconfig-mode: "0640"
write-kubeconfig-group: exedev
EOF
Move Traefik to port 8000 (the VM's main URL), drop port 443 (the proxy
does HTTPS), and trust the proxy's X-Forwarded-* headers, with a
HelmChartConfig:
sudo mkdir -p /var/lib/rancher/k3s/server/manifests
sudo tee /var/lib/rancher/k3s/server/manifests/traefik-config.yaml >/dev/null <<'EOF'
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: traefik
namespace: kube-system
spec:
valuesContent: |-
ports:
web:
exposedPort: 8000
forwardedHeaders:
insecure: true
websecure:
expose:
default: false
EOF
Only the proxy and the VM itself can reach port 8000, so trusting those headers is safe.
Install k3s#
The official installer, pinned to the tested version, sets up a k3s
systemd service plus kubectl and crictl:
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.2+k3s1 sh -
mkdir -p ~/.kube && ln -sf /etc/rancher/k3s/k3s.yaml ~/.kube/config
until kubectl get node k3s >/dev/null 2>&1; do sleep 2; done
kubectl wait --for=condition=Ready node/k3s --timeout=120s
The node is Ready within a minute. Then wait for Traefik (its Helm job may fail once or twice first; that's normal):
until kubectl -n kube-system get deploy/traefik >/dev/null 2>&1; do sleep 5; done
kubectl -n kube-system rollout status deploy/traefik --timeout=300s
kubectl -n kube-system get svc traefik
ip -4 addr show cni0 | grep inet
Expect 8000: under the Service's PORT(S), and 10.244.0.1/24 on cni0.
Install Helm#
cd /tmp
curl -fsSLO https://get.helm.sh/helm-v4.3.2-linux-amd64.tar.gz
curl -fsSL https://get.helm.sh/helm-v4.3.2-linux-amd64.tar.gz.sha256sum | sha256sum -c
tar xzf helm-v4.3.2-linux-amd64.tar.gz linux-amd64/helm
sudo install linux-amd64/helm /usr/local/bin/helm
rm -rf linux-amd64 helm-v4.3.2-linux-amd64.tar.gz*
cd ~
helm version
sha256sum prints OK.
Deploy the sample app#
mkdir -p ~/k3s
cat > ~/k3s/whoami.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: whoami
spec:
replicas: 2
selector:
matchLabels:
app: whoami
template:
metadata:
labels:
app: whoami
spec:
containers:
- name: whoami
image: traefik/whoami:v1.11
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: whoami
spec:
selector:
app: whoami
ports:
- port: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: whoami
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: whoami
port:
number: 80
EOF
kubectl apply -f ~/k3s/whoami.yaml
kubectl rollout status deploy/whoami --timeout=120s
curl -sf --retry 10 --retry-delay 2 localhost:8000/ | head -3
printf 'Open https://%s.exe.xyz/\n' "$(hostname)"
curl prints Hostname: whoami-....
Set up nightly backups#
A live, checked copy of the SQLite datastore, plus the server token (needed to decrypt it), certificates, config and all local-path volumes. Kept for two weeks:
mkdir -p ~/k3s/backups && chmod 700 ~/k3s/backups
cat > ~/k3s/backup.sh <<'EOF'
#!/bin/sh
# Back up k3s: datastore, token, certificates, config and volumes. Run as root.
set -eu
umask 077
out=/home/exedev/k3s/backups
k=/var/lib/rancher/k3s
work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT
mkdir -p "$work/server/db" "$work/etc"
sqlite3 "$k/server/db/state.db" ".backup '$work/server/db/state.db'"
test "$(sqlite3 "$work/server/db/state.db" 'PRAGMA integrity_check;')" = ok
cp -a "$k/server/token" "$k/server/tls" "$work/server/"
cp -a /etc/rancher/k3s/config.yaml "$k/server/manifests/traefik-config.yaml" "$work/etc/"
mkdir -p "$k/storage" && cp -a "$k/storage" "$work/"
file="$out/k3s-$(date +%Y%m%d-%H%M%S).tar.gz"
tar -C "$work" -czf "$file.tmp" .
mv "$file.tmp" "$file"
chown exedev:exedev "$file"
find "$out" -name 'k3s-*.tar.gz' -mtime +14 -delete
EOF
chmod 755 ~/k3s/backup.sh
sudo tee /etc/systemd/system/k3s-backup.service >/dev/null <<'EOF'
[Unit]
Description=Back up k3s
After=k3s.service
[Service]
Type=oneshot
ExecStart=/home/exedev/k3s/backup.sh
EOF
sudo tee /etc/systemd/system/k3s-backup.timer >/dev/null <<'EOF'
[Unit]
Description=Back up k3s nightly
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now k3s-backup.timer
sudo systemctl start k3s-backup.service
ls -l ~/k3s/backups
tar -tzf ~/k3s/backups/k3s-*.tar.gz | grep -E 'state.db|token$'
Expect one archive listing ./server/db/state.db and ./server/token.
Volumes are copied live; for a database in the cluster, also dump it
(pg_dump, say) into its volume before the backup runs.
Back up and restore
Back up now with sudo ~/k3s/backup.sh. Archives contain the token and
every Secret: keep them private, and keep a copy off the VM. On your
computer:
scp -r my-k3s.exe.xyz:k3s/backups .
Keep manifests and charts in git too.
Restore#
On a new VM, run Install by hand first and copy the archive into
~/k3s/backups. Then, on the VM, with your archive's name (the current
datastore and volumes are set aside, not deleted):
backup=~/k3s/backups/k3s-YYYYMMDD-HHMMSS.tar.gz
k=/var/lib/rancher/k3s
work=$(sudo mktemp -d)
sudo tar -C "$work" -xzpf "$backup" && sudo test -s "$work/server/db/state.db" &&
sudo systemctl stop k3s-backup.timer k3s &&
sudo /usr/local/bin/k3s-killall.sh >/dev/null &&
stamp=$(date +%s) &&
sudo mv "$k/server/db" "$k/server/db.before-restore-$stamp" &&
sudo mv "$k/storage" "$k/storage.before-restore-$stamp" &&
sudo mkdir -p "$k/server/db" &&
sudo cp "$work/server/db/state.db" "$k/server/db/" &&
sudo cp "$work/server/token" "$k/server/token" &&
sudo rm -rf "$k/server/tls" && sudo cp -a "$work/server/tls" "$k/server/" &&
sudo cp -a "$work/storage" "$k/" &&
sudo systemctl start k3s k3s-backup.timer &&
sudo rm -rf "$work"
kubectl get nodes
kubectl get pods -A
In a minute or two your workloads and volumes are back as of the backup.
Then sudo rm -rf the *.before-restore-* directories.
Update k3s and Helm
Re-run the installer with a newer version from k3s releases, one minor version at a time (1.37 → 1.38 → 1.39). Config, data and workloads stay. On the VM:
sudo ~/k3s/backup.sh
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.2+k3s1 sh - # the new version here
kubectl get nodes
Pods keep running while k3s restarts. Then check
kubectl -n kube-system get svc traefik still shows port 8000.
For Helm, repeat Install Helm with a version from Helm releases.
Uninstall
Removes k3s and all cluster data, including volumes; keeps the VM and
~/k3s/backups. On the VM:
sudo systemctl disable --now k3s-backup.timer
sudo /usr/local/bin/k3s-uninstall.sh
Or delete the VM: ssh exe.dev rm my-k3s.
Troubleshooting
On the VM:
kubectl get nodes
kubectl get pods -A
kubectl describe pod <name> -n <namespace> # the Events at the bottom
sudo journalctl -u k3s --since '10 min ago' --no-pager | tail -50
- Image pulls or DNS fail (
Try again) on a k3s installed some other way?ip -4 addr show cni0shows10.42.0.1: pods clash with the VM's network. Networks can't be changed in place; runsudo /usr/local/bin/k3s-uninstall.sh(deletes the cluster) and follow Install by hand. - kubectl: permission denied on
k3s.yaml? Add thewrite-kubeconfig-*lines toconfig.yaml, thensudo systemctl restart k3s. - helm: cluster unreachable,
localhost:8080? Rerun theln -sfline from Install k3s. 404 page not foundat the VM's URL? No Ingress matches; checkkubectl get ingress -A.- Something else, or nothing, at the URL? Check Traefik shows
8000:(kubectl -n kube-system get svc traefik) and/var/lib/rancher/k3s/server/manifests/traefik-config.yamlexists. - LoadBalancer stuck at
<pending>? Another Service has that port (kubectl get svc -A). ImagePullBackOffon an imported image? Usedocker.io/library/myapp:v1withimagePullPolicy: Never; checksudo k3s crictl images.- A VM copy shows a second,
NotReadynode? The original lackednode-name: k3s. Pin it at install; renaming a node later strands its local-path volumes.
Frequently asked questions#
Does Kubernetes work on exe.dev?
Yes. k3s runs on a standard VM with no kernel changes. The one catch: the VM's network is 10.42.0.0/16, k3s's default pod range, so this recipe moves pods and services to 10.244.0.0/16 and 10.96.0.0/16.
Is the Kubernetes API on the internet?
No. Only HTTPS (through the proxy) and SSH reach a VM, so port 6443 is closed to the world. Use kubectl on the VM or through an SSH tunnel.
What size VM do I need?
The default is enough to start: our 2-CPU, 8 GB test VM used about 1 GB of memory with k3s, Traefik and a sample app. Need more? Copy to a bigger VM: ssh exe.dev cp my-k3s --memory=16GB --disk=100GB.
Can I add more nodes?
Probably, over Tailscale, but we haven't tested it; see Add nodes. A bigger single VM is usually simpler.
Why k3s rather than minikube or kind?
kind and minikube run Kubernetes inside Docker, which suits a laptop. k3s runs directly under systemd, with ingress, storage and a load balancer built in. It's CNCF-certified, so manifests and charts work unchanged.
Can I run production on this?
Small internal services, if some downtime is acceptable. There's no high availability or managed control plane. It's best for dev clusters, CI, previews and learning.
Is k3s free?
Yes, under Apache-2.0, as are Helm and the local-path provisioner; Traefik is MIT. You pay for the VM; see exe.dev plans.
Updated Nov 16, 2026 · tested end to end Nov 16, 2026 · Markdown · manifest.json · Something wrong? Tell us