Skip to content

Run Kubernetes with k3s

A real Kubernetes cluster on one VM, private until you say otherwise.

Deploy on exe.dev

Shelley handles the setup. About 10 minutes.

Prefer a terminal?
curl -fsSL https://exe-staging.dev/deploy/k3s/prompt.txt | ssh exe.dev new --prompt=/dev/stdin

k3s is a CNCF-certified Kubernetes in one binary, with ingress (Traefik), a load balancer, local storage and CoreDNS built in. Here it runs under systemd on an exe.dev VM with a real disk: a cluster for learning, CI, Helm chart tests, or one copy per pull request.

What exe.dev adds: the Kubernetes API never faces the internet (you use kubectl over SSH). Ingress routes get HTTPS and exe.dev login from the HTTPS proxy, with no cert-manager or auth proxy, and apps see the caller in X-ExeDev-Email. Pods reach the LLM gateway and other integrations without API keys. Shelley, the agent on the VM, can run kubectl for you.

What you get#

  • k3s v1.37.2+k3s1 running as a systemd service, with pod and service networks that don't clash with the VM's own.
  • kubectl and Helm working for exedev without sudo.
  • Traefik answering on port 8000, with a sample app at the VM's private URL that shows your exe.dev email.
  • PersistentVolumeClaims backed by the VM's disk through the local-path provisioner.
  • Nightly backups of the cluster state, server token and volumes, with the first one checked.
What's running, and where
Private https://my-k3s.exe.xyz/ Traefik, serving your Ingress routes (a sample app to start)
Private https://my-k3s.exe.xyz:9898/ Any LoadBalancer Service on a port from 3000 to 9999
Private ssh -L 16443:localhost:6443 my-k3s.exe.xyz The Kubernetes API, for kubectl on your computer, over SSH only
my-k3s.exe.xyz exe.dev VM
k3s v1.37.2+k3s1 Kubernetes 1.37 with containerd, flannel, CoreDNS and a SQLite datastore, under systemd /var/lib/rancher/k3s
Traefik 3 k3s's bundled ingress controller, moved to port 8000 for exe.dev's HTTPS proxy :8000
local-path provisioner PersistentVolumes as directories on the VM's disk /var/lib/rancher/k3s/storage
Helm 4.3 The Kubernetes package manager
Nightly backup Cluster state, token and volumes; 14 nights kept ~/k3s/backups

Deploy with Shelley#

Click Deploy and Shelley will set up k3s on a new VM, then check that it works. No config-file wrangling required.

Prefer a terminal?
curl -fsSL \
  https://exe-staging.dev/deploy/k3s/prompt.txt \
  | ssh exe.dev new --prompt=/dev/stdin

When it's done, run this on your computer (only you can):

ssh my-k3s.exe.xyz cat /etc/rancher/k3s/k3s.yaml | sed 's/:6443/:16443/' > my-k3s.kubeconfig   # then KUBECONFIG=my-k3s.kubeconfig
ssh -N -L 16443:localhost:6443 my-k3s.exe.xyz
What Shelley is told
Set up k3s on this VM by following the exe.dev recipe at https://exe-staging.dev/deploy/k3s.md from top to bottom.

The recipe's example VM name is my-k3s. This VM's name is the output of `hostname`: use it wherever the recipe writes my-k3s. Skip the step that creates the VM; you are on it.

You are done when all of these are true. Check each one yourself:
- k3s v1.37.2+k3s1 running as a systemd service, with pod and service networks that don't clash with the VM's own.
- kubectl and Helm working for `exedev` without sudo.
- Traefik answering on port 8000, with a sample app at the VM's private URL that shows your exe.dev email.
- PersistentVolumeClaims backed by the VM's disk through the local-path provisioner.
- Nightly backups of the cluster state, server token and volumes, with the first one checked.

Read the whole recipe, including the closed details, and follow
"Install by hand" from top to bottom: the k3s config, the Traefik
config, k3s itself, Helm, the sample app, and the backup timer. Use the
pinned versions as written. The walkthrough sections are for the owner
afterwards: don't install podinfo or other charts, add nodes, set up
Tailscale, copy the VM, or run the Restore, Update or Uninstall
instructions during setup. Leave the VM private and do not reboot.
Check `kubectl get nodes` shows one Ready node named k3s, every pod in
kube-system is Running or Completed, and `curl localhost:8000` returns
the whoami output. Finish with the VM's private URL and the two
commands that give the owner kubectl on their own computer.

Finish with the URLs I should open and these commands, which only I can run, from my own computer (with this VM's name in them):
  ssh my-k3s.exe.xyz cat /etc/rancher/k3s/k3s.yaml | sed 's/:6443/:16443/' > my-k3s.kubeconfig   # then KUBECONFIG=my-k3s.kubeconfig
  ssh -N -L 16443:localhost:6443 my-k3s.exe.xyz

Also at /deploy/k3s/prompt.txt. The recipe it follows is /deploy/k3s.md.

Try the cluster#

On your computer, ssh my-k3s.exe.xyz. Then, on the VM:

kubectl get nodes
kubectl get pods -A

Expect one node, k3s, Ready, and the system pods Running.

Open https://my-k3s.exe.xyz/. After exe.dev login you'll see the sample app, whoami, echoing your request, including X-Exedev-Email: you@example.com. Its manifest is ~/k3s/whoami.yaml; swap in your own app when ready.

Use kubectl from your computer#

Install kubectl 1.36-1.38. On your computer, fetch the kubeconfig, pointed at local port 16443:

mkdir -p ~/.kube
ssh my-k3s.exe.xyz cat /etc/rancher/k3s/k3s.yaml \
  | sed 's/127.0.0.1:6443/127.0.0.1:16443/' > ~/.kube/my-k3s.yaml
chmod 600 ~/.kube/my-k3s.yaml

Open the tunnel in one terminal, use it from another:

ssh -N -L 16443:localhost:6443 my-k3s.exe.xyz
export KUBECONFIG=~/.kube/my-k3s.yaml
kubectl get nodes

Helm, k9s and Lens use the same KUBECONFIG. The file is cluster-admin: guard it like an SSH key.

Put an app on the web#

Both routes are private (exe.dev login) until you share them.

Ingress, at https://my-k3s.exe.xyz/. Traefik listens on port 8000, where the proxy sends the VM's main URL. Leave out host in your rules, as the sample does, so they also match copies of the VM. Use path prefixes for several apps.

LoadBalancer, at https://my-k3s.exe.xyz:<port>/. k3s's ServiceLB opens the Service's port on the VM, and the proxy forwards ports 3000 to 9999:

apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  type: LoadBalancer
  selector:
    app: myapp
  ports:
    - port: 3000        # on the VM, and in the URL
      targetPort: 8080  # in the pod

NodePorts (30000-32767) aren't proxied.

Share with ssh exe.dev share add my-k3s alice@example.com (Sharing). Traefik passes X-ExeDev-Email and X-ExeDev-UserID through, so apps can authorize by email (Login with exe). Trust them only on requests from the proxy: other pods can call your Service directly with any headers.

Go public with ssh exe.dev share set-public my-k3s. That opens port 8000, so every Ingress route; ports 3000-9999 stay private. Undo with share set-private.

Keep data on the disk#

The default StorageClass, local-path, stores each PersistentVolumeClaim as a directory under /var/lib/rancher/k3s/storage, included in the nightly backup:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: local-path
  resources:
    requests:
      storage: 5Gi

Sizes aren't enforced; volumes share the VM's disk. Deleting a claim deletes its data.

Install charts with Helm#

Try podinfo, whose chart ships its own tests. On the VM:

helm install podinfo oci://ghcr.io/stefanprodan/charts/podinfo \
  --version 6.17.0 --namespace podinfo --create-namespace \
  --set service.type=LoadBalancer --wait
helm test podinfo -n podinfo

Tests end in Phase: Succeeded; the app is at https://my-k3s.exe.xyz:9898/. Remove it with helm uninstall podinfo -n podinfo. For your own chart: helm lint, helm install --wait, helm test, helm uninstall.

Run your own images#

No registry needed: build with Docker on the VM and import into k3s.

docker build -t myapp:v1 .
docker save myapp:v1 | sudo k3s ctr images import -

Reference it as docker.io/library/myapp:v1 with imagePullPolicy: Never, and bump the tag on each build.

For a private registry, put credentials in /etc/rancher/k3s/registries.yaml and sudo systemctl restart k3s. For ghcr.io, with a read:packages-only token:

configs:
  ghcr.io:
    auth:
      username: your-github-user
      password: ghp_your_read_only_token

More in k3s's private registry docs. To keep that token off the VM, exe.dev's container registry integrations mint short-lived pull tokens (untested with k3s).

Call APIs without keys in Secrets#

Pods use the VM's network, so they reach its integrations. Point OpenAI- or Anthropic-compatible clients at https://llm.int.exe.xyz/v1 with no API key (LLM gateway). For other APIs, an HTTP proxy integration adds the credential at the edge, so it's never in a Secret or a backup. See agent secrets. Any pod can use them, so run only workloads you trust.

A cluster per pull request#

A copy of the VM is the same cluster, apps and volumes included, at a new URL. On your computer:

ssh exe.dev cp my-k3s my-k3s-pr-42

About a minute later https://my-k3s-pr-42.exe.xyz/ is up. Deploy the branch there (helm upgrade, kubectl set image), share the link, and ssh exe.dev rm my-k3s-pr-42 after merging. This relies on the pinned node name k3s; see Configure k3s.

For CI, keep one VM as a warm template and copy it per job; see GitHub Actions and CI, or ask Shelley to script it.

Add nodes#

Warning

Untested on exe.dev so far.

VMs share no private network and all have 10.42.0.42 on eth0, so run flannel over Tailscale (the Tailscale integration joins without an auth key on the VM). On the server, add to /etc/rancher/k3s/config.yaml, then sudo systemctl restart k3s:

node-ip: 100.x.y.z          # tailscale ip -4
node-external-ip: 100.x.y.z
flannel-iface: tailscale0
tls-san: [100.x.y.z]

On each agent, /etc/rancher/k3s/config.yaml:

server: https://100.x.y.z:6443   # the server's Tailscale address
token: K10...                    # sudo cat /var/lib/rancher/k3s/server/node-token on the server
node-ip: 100.a.b.c               # this agent's Tailscale address
node-external-ip: 100.a.b.c
flannel-iface: tailscale0

Then curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.2+k3s1 sh -s - agent. Local-path volumes stay on the node that made them. k3s also has an experimental Tailscale integration.

Limits#

  • One node, no control-plane failover.
  • No cloud load balancer. ServiceLB opens ports on the VM; only HTTP on 8000 and 3000-9999 reaches the outside. Raw TCP/UDP needs SSH tunnels or Tailscale.
  • No network block storage. Volumes are directories on one disk.
  • TLS is the proxy's, so no cert-manager for exe.xyz names. For your own domain, see custom domains.

The nuts and bolts#

Shelley's setup steps, and your maintenance reference.

Install by hand

Create a VM#

On your computer (pick another name if my-k3s is taken):

ssh exe.dev new --name=my-k3s
ssh my-k3s.exe.xyz

Everything else runs on the VM, as exedev.

Configure k3s#

Three settings matter on exe.dev:

  • Networks. The VM's own network is 10.42.0.0/16, k3s's default pod network. Move pods and services to kubeadm's default ranges.
  • Node name. Pinned to k3s, so VM copies keep their pods and volumes.
  • Kubeconfig. Group-readable, so you and Shelley need no sudo.
sudo mkdir -p /etc/rancher/k3s
sudo tee /etc/rancher/k3s/config.yaml >/dev/null <<'EOF'
cluster-cidr: 10.244.0.0/16
service-cidr: 10.96.0.0/16
cluster-dns: 10.96.0.10
node-name: k3s
write-kubeconfig-mode: "0640"
write-kubeconfig-group: exedev
EOF

Move Traefik to port 8000 (the VM's main URL), drop port 443 (the proxy does HTTPS), and trust the proxy's X-Forwarded-* headers, with a HelmChartConfig:

sudo mkdir -p /var/lib/rancher/k3s/server/manifests
sudo tee /var/lib/rancher/k3s/server/manifests/traefik-config.yaml >/dev/null <<'EOF'
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ports:
      web:
        exposedPort: 8000
        forwardedHeaders:
          insecure: true
      websecure:
        expose:
          default: false
EOF

Only the proxy and the VM itself can reach port 8000, so trusting those headers is safe.

Install k3s#

The official installer, pinned to the tested version, sets up a k3s systemd service plus kubectl and crictl:

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.2+k3s1 sh -
mkdir -p ~/.kube && ln -sf /etc/rancher/k3s/k3s.yaml ~/.kube/config
until kubectl get node k3s >/dev/null 2>&1; do sleep 2; done
kubectl wait --for=condition=Ready node/k3s --timeout=120s

The node is Ready within a minute. Then wait for Traefik (its Helm job may fail once or twice first; that's normal):

until kubectl -n kube-system get deploy/traefik >/dev/null 2>&1; do sleep 5; done
kubectl -n kube-system rollout status deploy/traefik --timeout=300s
kubectl -n kube-system get svc traefik
ip -4 addr show cni0 | grep inet

Expect 8000: under the Service's PORT(S), and 10.244.0.1/24 on cni0.

Install Helm#

cd /tmp
curl -fsSLO https://get.helm.sh/helm-v4.3.2-linux-amd64.tar.gz
curl -fsSL https://get.helm.sh/helm-v4.3.2-linux-amd64.tar.gz.sha256sum | sha256sum -c
tar xzf helm-v4.3.2-linux-amd64.tar.gz linux-amd64/helm
sudo install linux-amd64/helm /usr/local/bin/helm
rm -rf linux-amd64 helm-v4.3.2-linux-amd64.tar.gz*
cd ~
helm version

sha256sum prints OK.

Deploy the sample app#

mkdir -p ~/k3s
cat > ~/k3s/whoami.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami
spec:
  replicas: 2
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: traefik/whoami:v1.11
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: whoami
spec:
  selector:
    app: whoami
  ports:
    - port: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: whoami
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: whoami
                port:
                  number: 80
EOF
kubectl apply -f ~/k3s/whoami.yaml
kubectl rollout status deploy/whoami --timeout=120s
curl -sf --retry 10 --retry-delay 2 localhost:8000/ | head -3
printf 'Open https://%s.exe.xyz/\n' "$(hostname)"

curl prints Hostname: whoami-....

Set up nightly backups#

A live, checked copy of the SQLite datastore, plus the server token (needed to decrypt it), certificates, config and all local-path volumes. Kept for two weeks:

mkdir -p ~/k3s/backups && chmod 700 ~/k3s/backups
cat > ~/k3s/backup.sh <<'EOF'
#!/bin/sh
# Back up k3s: datastore, token, certificates, config and volumes. Run as root.
set -eu
umask 077
out=/home/exedev/k3s/backups
k=/var/lib/rancher/k3s
work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT
mkdir -p "$work/server/db" "$work/etc"
sqlite3 "$k/server/db/state.db" ".backup '$work/server/db/state.db'"
test "$(sqlite3 "$work/server/db/state.db" 'PRAGMA integrity_check;')" = ok
cp -a "$k/server/token" "$k/server/tls" "$work/server/"
cp -a /etc/rancher/k3s/config.yaml "$k/server/manifests/traefik-config.yaml" "$work/etc/"
mkdir -p "$k/storage" && cp -a "$k/storage" "$work/"
file="$out/k3s-$(date +%Y%m%d-%H%M%S).tar.gz"
tar -C "$work" -czf "$file.tmp" .
mv "$file.tmp" "$file"
chown exedev:exedev "$file"
find "$out" -name 'k3s-*.tar.gz' -mtime +14 -delete
EOF
chmod 755 ~/k3s/backup.sh

sudo tee /etc/systemd/system/k3s-backup.service >/dev/null <<'EOF'
[Unit]
Description=Back up k3s
After=k3s.service

[Service]
Type=oneshot
ExecStart=/home/exedev/k3s/backup.sh
EOF
sudo tee /etc/systemd/system/k3s-backup.timer >/dev/null <<'EOF'
[Unit]
Description=Back up k3s nightly

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now k3s-backup.timer
sudo systemctl start k3s-backup.service
ls -l ~/k3s/backups
tar -tzf ~/k3s/backups/k3s-*.tar.gz | grep -E 'state.db|token$'

Expect one archive listing ./server/db/state.db and ./server/token. Volumes are copied live; for a database in the cluster, also dump it (pg_dump, say) into its volume before the backup runs.

Back up and restore

Back up now with sudo ~/k3s/backup.sh. Archives contain the token and every Secret: keep them private, and keep a copy off the VM. On your computer:

scp -r my-k3s.exe.xyz:k3s/backups .

Keep manifests and charts in git too.

Restore#

On a new VM, run Install by hand first and copy the archive into ~/k3s/backups. Then, on the VM, with your archive's name (the current datastore and volumes are set aside, not deleted):

backup=~/k3s/backups/k3s-YYYYMMDD-HHMMSS.tar.gz
k=/var/lib/rancher/k3s
work=$(sudo mktemp -d)
sudo tar -C "$work" -xzpf "$backup" && sudo test -s "$work/server/db/state.db" &&
sudo systemctl stop k3s-backup.timer k3s &&
sudo /usr/local/bin/k3s-killall.sh >/dev/null &&
stamp=$(date +%s) &&
sudo mv "$k/server/db" "$k/server/db.before-restore-$stamp" &&
sudo mv "$k/storage" "$k/storage.before-restore-$stamp" &&
sudo mkdir -p "$k/server/db" &&
sudo cp "$work/server/db/state.db" "$k/server/db/" &&
sudo cp "$work/server/token" "$k/server/token" &&
sudo rm -rf "$k/server/tls" && sudo cp -a "$work/server/tls" "$k/server/" &&
sudo cp -a "$work/storage" "$k/" &&
sudo systemctl start k3s k3s-backup.timer &&
sudo rm -rf "$work"
kubectl get nodes
kubectl get pods -A

In a minute or two your workloads and volumes are back as of the backup. Then sudo rm -rf the *.before-restore-* directories.

Update k3s and Helm

Re-run the installer with a newer version from k3s releases, one minor version at a time (1.37 → 1.38 → 1.39). Config, data and workloads stay. On the VM:

sudo ~/k3s/backup.sh
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.2+k3s1 sh -   # the new version here
kubectl get nodes

Pods keep running while k3s restarts. Then check kubectl -n kube-system get svc traefik still shows port 8000.

For Helm, repeat Install Helm with a version from Helm releases.

Uninstall

Removes k3s and all cluster data, including volumes; keeps the VM and ~/k3s/backups. On the VM:

sudo systemctl disable --now k3s-backup.timer
sudo /usr/local/bin/k3s-uninstall.sh

Or delete the VM: ssh exe.dev rm my-k3s.

Troubleshooting

On the VM:

kubectl get nodes
kubectl get pods -A
kubectl describe pod <name> -n <namespace>   # the Events at the bottom
sudo journalctl -u k3s --since '10 min ago' --no-pager | tail -50
  • Image pulls or DNS fail (Try again) on a k3s installed some other way? ip -4 addr show cni0 shows 10.42.0.1: pods clash with the VM's network. Networks can't be changed in place; run sudo /usr/local/bin/k3s-uninstall.sh (deletes the cluster) and follow Install by hand.
  • kubectl: permission denied on k3s.yaml? Add the write-kubeconfig-* lines to config.yaml, then sudo systemctl restart k3s.
  • helm: cluster unreachable, localhost:8080? Rerun the ln -sf line from Install k3s.
  • 404 page not found at the VM's URL? No Ingress matches; check kubectl get ingress -A.
  • Something else, or nothing, at the URL? Check Traefik shows 8000: (kubectl -n kube-system get svc traefik) and /var/lib/rancher/k3s/server/manifests/traefik-config.yaml exists.
  • LoadBalancer stuck at <pending>? Another Service has that port (kubectl get svc -A).
  • ImagePullBackOff on an imported image? Use docker.io/library/myapp:v1 with imagePullPolicy: Never; check sudo k3s crictl images.
  • A VM copy shows a second, NotReady node? The original lacked node-name: k3s. Pin it at install; renaming a node later strands its local-path volumes.

Frequently asked questions#

Does Kubernetes work on exe.dev?

Yes. k3s runs on a standard VM with no kernel changes. The one catch: the VM's network is 10.42.0.0/16, k3s's default pod range, so this recipe moves pods and services to 10.244.0.0/16 and 10.96.0.0/16.

Is the Kubernetes API on the internet?

No. Only HTTPS (through the proxy) and SSH reach a VM, so port 6443 is closed to the world. Use kubectl on the VM or through an SSH tunnel.

What size VM do I need?

The default is enough to start: our 2-CPU, 8 GB test VM used about 1 GB of memory with k3s, Traefik and a sample app. Need more? Copy to a bigger VM: ssh exe.dev cp my-k3s --memory=16GB --disk=100GB.

Can I add more nodes?

Probably, over Tailscale, but we haven't tested it; see Add nodes. A bigger single VM is usually simpler.

Why k3s rather than minikube or kind?

kind and minikube run Kubernetes inside Docker, which suits a laptop. k3s runs directly under systemd, with ingress, storage and a load balancer built in. It's CNCF-certified, so manifests and charts work unchanged.

Can I run production on this?

Small internal services, if some downtime is acceptable. There's no high availability or managed control plane. It's best for dev clusters, CI, previews and learning.

Is k3s free?

Yes, under Apache-2.0, as are Helm and the local-path provisioner; Traefik is MIT. You pay for the VM; see exe.dev plans.

Updated Nov 16, 2026 · tested end to end Nov 16, 2026 · Markdown · manifest.json · Something wrong? Tell us