Charmed Kubernetes on Equinix Metal
As with any cloud supported by Juju, Charmed Kubernetes can be deployed and used on Equinix Metal. This document provides some extra information and an overlay to help get the most out of this cloud. For instructions on installing Juju itself, please see the latest Juju documentation.
Before installing
Equinix Metal has been added to the clouds Juju automatically knows about. To check, you can run the command:
juju list-clouds --all
If equinix does not appear in the list, your local Juju install probably just needs to
refresh its list of clouds. Run:
juju update-public-clouds
You should also add your credentials for this cloud. Use the interactive command:
juju add-credential equinix
…and follow the prompts to enter the information required (including the project id, and your auth token).
Bootstrapping
Bootstrap a Juju controller within one of the equinix regions.
For example, to select Dallas ‘da’ from the list of equinix-facilities
juju bootstrap equinix/da equinix-da \
--bootstrap-series focal\
--bootstrap-constraints arch=amd64\
--model-default image-stream=daily\
--model-default automatically-retry-hooks=true\
--model-default 'logging-config=<root>=DEBUG'\
--debug --verbose -vv
Installing
To deploy Charmed Kubernetes on Equinix Metal, it is also recommended to deploy some storage and to use Calico for networking. You can deploy and configure Charmed Kubernetes any way you like, but this example overlay will help you get started.
You can download the ceph-radosgw overlay here And download the equinix overlay here
It can then be installed with the command:
juju deploy charmed-kubernetes --overlay ./ceph-radosgw.yaml --overlay ./equinix-overlay.yaml
When the deployment has settled, remember to fetch the configuration file!
juju ssh kubernetes-control-plane/leader -- cat config > ~/.kube/config
You can check the status by running:
juju status
At this point, there will be error messages on the workers as the pods will not run until the Cloud Controller Manager has been run.
Post install
To use Kubernetes on Equinix Metal, you should now set up the Equinix Cloud Controller Manager.
While the deployment is in progress no pods will be able to spun up on the Kubernetes due to taints being set on each node. The taints will be removed once the Cloud Controller Manager (CCM) is enabled and the nodes are registered with the cloud control plane.
First, a Kubernetes secret has to be created, defining the variables for the CCM: Configuration of the CCM can be applied via the secret, See equinix-configuration for detalis.
cat <<EOY > secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: metal-cloud-config
namespace: kube-system
stringData:
cloud-sa.json: |
{
"apiKey": "<Metal API key>",
"projectID": "<Metal Project ID>",
"loadbalancer": "kube-vip://",
"metro": "da"
}
EOY
vim secret.yaml
kubectl create -f secret.yaml
The next steps are to confirm the version of the CCM to use:
export CCM_VERSION=3.5.0
kubectl apply -f https://github.com/equinix/cloud-provider-equinix-metal/releases/download/v${CCM_VERSION}/deployment.yaml
…enable premissions for the kube-vip loadbalancer:
kubectl apply -f https://kube-vip.io/manifests/rbac.yaml
… and deploy:
kubectl apply -f - <<EOY
apiVersion: apps/v1
kind: DaemonSet
metadata:
creationTimestamp: null
name: kube-vip-ds
namespace: kube-system
spec:
selector:
matchLabels:
name: kube-vip-ds
template:
metadata:
creationTimestamp: null
labels:
name: kube-vip-ds
spec:
containers:
- args:
- manager
env:
- name: vip_arp
value: "false"
- name: vip_interface
value: lo
- name: port
value: "6443"
- name: vip_cidr
value: "32"
- name: svc_enable
value: "true"
- name: vip_startleader
value: "false"
- name: vip_addpeerstolb
value: "true"
- name: annotation
value: metal.equinix.com
- name: bgp_enable
value: "true"
- name: bgp_routerid
- name: bgp_as
value: "65000"
- name: bgp_peeraddress
- name: bgp_peerpass
- name: bgp_peeras
value: "65000"
- name: vip_address
image: plndr/kube-vip:v0.3.6
imagePullPolicy: Always
name: kube-vip
resources: {}
securityContext:
capabilities:
add:
- NET_ADMIN
- NET_RAW
- SYS_TIME
hostNetwork: true
serviceAccountName: kube-vip
updateStrategy: {}
status:
currentNumberScheduled: 0
desiredNumberScheduled: 0
numberMisscheduled: 0
numberReady: 0
EOY
Note: in some Equinix Metal facilities it is required to define a static route on each Kubernetes Worker node to allow the traffic to the workloads exposed via the Load Balancer to go via proper gateway:
juju exec --application kubernetes-worker,kubernetes-control-plane '
apt install jq -y
GATEWAY_IP=$(curl https://metadata.platformequinix.com/metadata | jq -r ".network.addresses[] | select(.public == false) | .gateway")
PEERS=$(curl https://metadata.platformequinix.com/metadata | jq -r ".bgp_neighbors[0].peer_ips[]")
for i in ${PEERS}; do
ip route add ${i} via $GATEWAY_IP
done
Using load balancers
With the cloud load balancer capabilities enabled, actions which invoke a loadbalancer in Kubernetes will automatically trigger creation of the ElasticIP in the Metal cloud and associate it with the KubeVIP service, simultaneously adjusting BGP tables in the cloud and forward the traffic to Kubernetes nodes. This can be demonstrated with a simple application. Here we will create a simple application and scale it to five pods:
kubectl create deployment hello-world --image=gcr.io/google-samples/node-hello:1.0
kubectl scale deployment hello-world --replicas=5
You can verify that the application and replicas have been created with:
kubectl get deployments hello-world
Which should return output similar to:
NAME READY UP-TO-DATE AVAILABLE AGE
hello-world 5/5 5 5 2m38s
To create a LoadBalancer, the application should now be exposed as a service:
kubectl expose deployment hello-world --type=LoadBalancer --name=hello --port 8080
To check that the service is running correctly:
kubectl get service hello
…which should return output similar to:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello LoadBalancer 10.152.183.136 202.49.242.3 8080:32662/TCP 2m
You can see that the External IP is now in front of the five endpoints of the example deployment. You can test the ingress address:
curl http://202.49.242.3:8080
Hello Kubernetes!