📄 Deploying Talos and Kubernetes with Cluster-API on Equinix Metal

In this guide we will launch a highly-available three Node Kubernetes cluster on Equinix Metal using Talos as the Node OS, as well as bootstrap, and controlPlane provider for Cluster-API.

What is Cluster-API?

Cluster API is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters.

What is Talos?

Talos is a modern OS designed to be secure, immutable, and minimal.

What is Equinix Metal?

A globally-available bare metal “as-a-service” that can be deployed and interconnected in minutes.

The folks over at Equinix Metal have a wonderful heart for supporting Open Source communities.

Why is this important?

In general: Orchestrating a container based OS such as Talos (Flatcar, Fedora CoreOS, or RancherOS) shifts focus from the Nodes to the workloads. In terms of Talos: Currently the documentation for running an OS such as Talos in Equinix Metal for Kubernetes with Cluster-API is not so well documented and therefore inaccessible. It's important to fill in the gaps of knowledge.

What you'll need for this guide:

• talosctl
• kubectl
• packet-cli
• the ID and API token of existing Equinix Metal project
• an existing Kubernetes cluster with a public IP (such as kind, minikube, or a cluster already on Equinix Metal)

In order to talk to Equinix Metal, we'll export environment variables to configure resources and talk via packet-cli.

Set the correct project to create and manage resources in:

read -p 'PACKET_PROJECT_ID: ' PACKET_PROJECT_ID

The API key for your account or project:

read -p 'PACKET_API_KEY: ' PACKET_API_KEY

Export the variables to be accessible from packet-cli and clusterctl later on:

export PACKET_PROJECT_ID PACKET_API_KEY PACKET_TOKEN=$PACKET_API_KEY

In the existing cluster, a public LoadBalancer IP will be needed. I have already installed nginx-ingress in this cluster, which has got a Service with the cluster's elastic IP. We'll need this IP address later for use in booting the servers. If you have set up your existing cluster differently, it'll just need to be an IP that we can use.

export LOAD_BALANCER_IP="$(kubectl -n nginx-ingress get svc nginx-ingress-ingress-nginx-controller -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')"

Install Talos providers for Cluster-API bootstrap and controlplane in your existing cluster:

clusterctl init -b talos -c talos -i packet

This will install Talos's bootstrap and controlPlane controllers as well as the Packet / Equinix Metal infrastructure provider.

Important note:

• the bootstrap-talos controller in the cabpt-system namespace must be running a version greater than v0.2.0-alpha.8. The version can be displayed in with clusterctl upgrade plan when it's installed.

Currently, since Equinix Metal have not yet added support for Talos, it is necessary to install Matchbox to boot the servers (There is an issue in progress and feedback for adding support).

What is Matchbox?

Matchbox is a service that matches bare-metal machines to profiles that PXE boot and provision clusters.

Here is the manifest for a basic matchbox installation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: matchbox
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      name: matchbox
  template:
    metadata:
      labels:
        name: matchbox
    spec:
      containers:
        - name: matchbox
          image: quay.io/poseidon/matchbox:v0.9.0
          env:
            - name: MATCHBOX_ADDRESS
              value: "0.0.0.0:8080"
            - name: MATCHBOX_LOG_LEVEL
              value: "debug"
          ports:
            - name: http
              containerPort: 8080
          livenessProbe:
            initialDelaySeconds: 5
            httpGet:
              path: /
              port: 8080
          resources:
            requests:
              cpu: 30m
              memory: 20Mi
            limits:
              cpu: 50m
              memory: 50Mi
          volumeMounts:
            - name: data
              mountPath: /var/lib/matchbox
            - name: assets
              mountPath: /var/lib/matchbox/assets
      volumes:
        - name: data
          hostPath:
            path: /var/local/matchbox/data
        - name: assets
          hostPath:
            path: /var/local/matchbox/assets
---
apiVersion: v1
kind: Service
metadata:
  name: matchbox
  annotations:
    metallb.universe.tf/allow-shared-ip: nginx-ingress
spec:
  type: LoadBalancer
  selector:
    name: matchbox
  ports:
    - name: http
      protocol: TCP
      port: 8080
      targetPort: 8080

Save it as matchbox.yaml

The manifests above were inspired by the manifests in the matchbox repo. For production it might be wise to use:

• an Ingress with full TLS
• a ReadWriteMany storage provider instead hostPath for scaling

With the manifests ready to go, we'll install Matchbox into the matchbox namespace on the existing cluster with the following commands:

kubectl create ns matchbox
kubectl -n matchbox apply -f ./matchbox.yaml

You may need to patch the Service.spec.externalIPs to have an IP to access it from if one is not populated:

kubectl -n matchbox patch \
  service matchbox \
  -p "{\"spec\":{\"externalIPs\":[\"$LOAD_BALANCER_IP\"]}}"

Once the pod is live, We'll need to create a directory structure for storing Talos boot assets:

kubectl -n matchbox exec -it \
  deployment/matchbox -- \
  mkdir -p /var/lib/matchbox/{profiles,groups} /var/lib/matchbox/assets/talos

Inside the Matchbox container, we'll download the Talos boot assets for Talos version 0.8.1 into the assets folder:

kubectl -n matchbox exec -it \
  deployment/matchbox -- \
  wget -P /var/lib/matchbox/assets/talos \
  https://github.com/talos-systems/talos/releases/download/v0.8.1/initramfs-amd64.xz \
  https://github.com/talos-systems/talos/releases/download/v0.8.1/vmlinuz-amd64

Now that the assets have been downloaded, run a checksum against them to verify:

kubectl -n matchbox exec -it \
  deployment/matchbox -- \
  sh -c "cd /var/lib/matchbox/assets/talos && \
    wget -O- https://github.com/talos-systems/talos/releases/download/v0.8.1/sha512sum.txt 2> /dev/null \
    | sed 's,_out/,,g' \
    | grep 'initramfs-amd64.xz\|vmlinuz-amd64' \
    | sha512sum -c -"

Since there's only one Pod in the Matchbox deployment, we'll export it's name to copy files into it:

export MATCHBOX_POD_NAME=$(kubectl -n matchbox get pods -l name=matchbox -o=jsonpath='{.items[0].metadata.name}')

Profiles in Matchbox are JSON configurations for how the servers should boot, where from, and their kernel args. Save this file as profile-default-amd64.json

{
  "id": "default-amd64",
  "name": "default-amd64",
  "boot": {
    "kernel": "/assets/talos/vmlinuz-amd64",
    "initrd": [
      "/assets/talos/initramfs-amd64.xz"
    ],
    "args": [
      "initrd=initramfs-amd64.xz",
      "init_on_alloc=1",
      "init_on_free=1",
      "slub_debug=P",
      "pti=on",
      "random.trust_cpu=on",
      "console=tty0",
      "console=ttyS1,115200n8",
      "slab_nomerge",
      "printk.devkmsg=on",
      "talos.platform=packet",
      "talos.config=none"
    ]
  }
}

Groups in Matchbox are a way of letting servers pick up profiles based on selectors. Save this file as group-default-amd64.json

{
  "id": "default-amd64",
  "name": "default-amd64",
  "profile": "default-amd64",
  "selector": {
    "arch": "amd64"
  }
}

We'll copy the profile and group into their respective folders:

kubectl -n matchbox \
  cp ./profile-default-amd64.json \
  $MATCHBOX_POD_NAME:/var/lib/matchbox/profiles/default-amd64.json
kubectl -n matchbox \
  cp ./group-default-amd64.json \
  $MATCHBOX_POD_NAME:/var/lib/matchbox/groups/default-amd64.json

List the files to validate that they were written correctly:

kubectl -n matchbox exec -it \
  deployment/matchbox -- \
  sh -c 'ls -alh /var/lib/matchbox/*/'

curl http://$LOAD_BALANCER_IP:8080

#!talos

packet-cli device create \
  --hostname talos-pxe-boot-test-1 \
  --plan c1.small.x86 \
  --facility sjc1 \
  --operating-system custom_ipxe \
  --project-id "$PACKET_PROJECT_ID" \
  --ipxe-script-url "http://$LOAD_BALANCER_IP:8080/ipxe?arch=amd64" \
  --userdata-file=./userdata.txt

kubectl -n matchbox logs deployment/matchbox -f --tail=100

In order to manage Talos Nodes outside of Kubernetes, we need to create and set up configuration to use.

Create the directory for the config:

mkdir -p $HOME/.talos

Discover the IP for the first controlPlane:

export TALOS_ENDPOINT=$(kubectl -n "$CLUSTER_NAME" \
  get machines \
  $(kubectl -n "$CLUSTER_NAME" \
    get machines -l cluster.x-k8s.io/control-plane='' \
    --no-headers --output=jsonpath='{.items[0].metadata.name}') \
    -o=jsonpath="{.status.addresses[?(@.type=='ExternalIP')].address}" | awk '{print $2}')

Fetch the talosconfig from the existing cluster:

kubectl get talosconfig \
  -n $CLUSTER_NAME \
  -l cluster.x-k8s.io/cluster-name=$CLUSTER_NAME \
  -o yaml -o jsonpath='{.items[0].status.talosConfig}' > $HOME/.talos/"$CLUSTER_NAME"-management-plane-talosconfig.yaml

Write in the configuration the endpoint IP and node IP:

talosctl \
  --talosconfig $HOME/.talos/"$CLUSTER_NAME"-management-plane-talosconfig.yaml \
  config endpoint $TALOS_ENDPOINT
talosctl \
  --talosconfig $HOME/.talos/"$CLUSTER_NAME"-management-plane-talosconfig.yaml \
  config node $TALOS_ENDPOINT

Now that the talosconfig has been written, try listing all containers:

<<cluster-config-env-name>>
# removing ip; omit ` | sed ...` for regular use
talosctl --talosconfig $HOME/.talos/"$CLUSTER_NAME"-management-plane-talosconfig.yaml containers | sed -r 's/(\b[0-9]{1,3}\.){3}[0-9]{1,3}\b'/"x.x.x.x      "/

Here's the containers running on this particular node, in containerd (not k8s related):

NODE            NAMESPACE   ID         IMAGE                                  PID    STATUS
x.x.x.x         system      apid       talos/apid                             3046   RUNNING
x.x.x.x         system      etcd       gcr.io/etcd-development/etcd:v3.4.14   3130   RUNNING
x.x.x.x         system      networkd   talos/networkd                         2879   RUNNING
x.x.x.x         system      routerd    talos/routerd                          2888   RUNNING
x.x.x.x         system      timed      talos/timed                            2976   RUNNING
x.x.x.x         system      trustd     talos/trustd                           3047   RUNNING

Tearing down the entire cluster and resources associated with it, can be achieved by

1. Deleting the cluster:

kubectl -n "$CLUSTER_NAME" delete cluster "$CLUSTER_NAME"

ii. Deleting the namespace:

kubectl delete ns "$CLUSTER_NAME"

iii. Removing local configurations:

rm \
  $HOME/.talos/"$CLUSTER_NAME"-management-plane-talosconfig.yaml \
  $HOME/.kube/"$CLUSTER_NAME"

(always learning) how wonderful the Kubernetes community is

there are so many knowledgable individuals who are so ready for collaboration and adoption - it doesn't matter the SIG or group.

how modular Cluster-API is

Cluster-API components (bootstrap, controlPlane, core, infrastructure) can be swapped out and meshed together in very cool ways.

Integrating Talos into this project would not be possible without help from Andrew Rynhard (Talos Systems), huge thanks to him for reaching out for pairing and co-authoring.

• with the new cluster's controlPlane live and available for deployment, the iPXE server could be moved into that cluster - meaning that new servers boot from the cluster that they'll join, making it almost self-contained
• cluster configuration as based off of cluster-template.yaml from the cluster-api-provider-packet repo
• this post has been made to calebwoodine.nz, and is also available as an Org file

Hope you've enjoyed the output of this project! Thank you!