This post describes how to change TKGm control plane nodes resources, such as vCPU and RAM. In the previous post, I described how to increase resources for a worker node. This process was quite simple and straightforward and initially I had a tough time finding the right resource to edit as the control plane nodes use a different resource to provision the virtual machines.
Step 1. Change to the TKG management cluster context
kubectl config use-context tkg-mgmtStep 2. List VSphereMachineTemplate
kubectl get VSphereMachineTemplateStep 4. Make a copy of the current control plane VsphereMachineTemplate to a new file
kubectl get vspheremachinetemplates tkg-ssc-control-plane -o yaml > tkg-ssc-control-plane-new.yamlStep 5. Edit the new file and make the changes
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: VSphereMachineTemplate
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"infrastructure.cluster.x-k8s.io/v1alpha3","kind":"VSphereMachineTemplate","metadata":{"annotations":{},"name":"tkg-ssc-control-plane-new","namespace":"default"},"spec":{"template":{"spec":{"cloneMode":"fullClone","datacenter":"/home.local","datastore":"lun01","diskGiB":40,"folder":"/home.local/vm/tkg-vsphere-shared-services","memoryMiB":4096,"network":{"devices":[{"dhcp4":true,"networkName":"/home.local/network/tkg-mgmt"}]},"numCPUs":2,"resourcePool":"/home.local/host/cluster/Resources/tkg-vsphere-shared-services","server":"vcenter.vmwire.com","storagePolicyName":"","template":"/home.local/vm/Templates/ubuntu-2004-kube-v1.21.2+vmware.1"}}}}
creationTimestamp: "2021-11-11T07:33:37Z"
generation: 1
name: tkg-ssc-control-plane-new
namespace: default
ownerReferences:
- apiVersion: cluster.x-k8s.io/v1alpha3
kind: Cluster
name: tkg-ssc
uid: 9bd41852-38df-4d12-bb81-7b2bb35fdfa5
resourceVersion: "198053"
uid: 09b62ee7-6532-4bf6-8939-ef70a28bc65f
spec:
template:
spec:
cloneMode: fullClone
datacenter: /home.local
datastore: lun01
diskGiB: 40
folder: /home.local/vm/tkg-vsphere-shared-services
memoryMiB: 4096
network:
devices:
- dhcp4: true
networkName: /home.local/network/tkg-mgmt
numCPUs: 2
resourcePool: /home.local/host/cluster/Resources/tkg-vsphere-shared-services
server: vcenter.vmwire.com
storagePolicyName: ""
template: /home.local/vm/Templates/ubuntu-2004-kube-v1.21.2+vmware.1
I made changes to lines 6, 9, 26 and 31. I want to reduce the vCPU and RAM of the control plane nodes as these were over-provisioned by mistake.
Step 6. Apply the new VsphereMachineTemplate
kubectl apply -f tkg-ssc-control-plane-new.yamlStep 7. List all KubeadmControlPlane
kubectl get KubeadmControlPlane
NAME INITIALIZED API SERVER AVAILABLE VERSION REPLICAS READY UPDATED UNAVAILABLE
tkg-ssc-control-plane true true v1.21.2+vmware.1 1 1 1
Step 8. Edit the KubeadmControlPlane for the cluster
kubectl edit KubeadmControlPlane tkg-ssc-control-plane
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: KubeadmControlPlane
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"controlplane.cluster.x-k8s.io/v1alpha3","kind":"KubeadmControlPlane","metadata":{"annotations":{},"name":"tkg-ssc-control-plane","namespace":"default"},"spec":{"infrastructureTemplate":{"apiVersion":"infrastructure.cluster.x-k8s.io/v1alpha3","kind":"VSphereMachineTemplate","name":"tkg-ssc-control-plane"},"kubeadmConfigSpec":{"clusterConfiguration":{"apiServer":{"extraArgs":{"audit-log-maxage":"30","audit-log-maxbackup":"10","audit-log-maxsize":"100","audit-log-path":"/var/log/kubernetes/audit.log","audit-policy-file":"/etc/kubernetes/audit-policy.yaml","cloud-provider":"external","tls-cipher-suites":"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"},"extraVolumes":[{"hostPath":"/etc/kubernetes/audit-policy.yaml","mountPath":"/etc/kubernetes/audit-policy.yaml","name":"audit-policy"},{"hostPath":"/var/log/kubernetes","mountPath":"/var/log/kubernetes","name":"audit-logs"}],"timeoutForControlPlane":"8m0s"},"controllerManager":{"extraArgs":{"cloud-provider":"external","tls-cipher-suites":"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"}},"dns":{"imageRepository":"projects.registry.vmware.com/tkg","imageTag":"v1.8.0_vmware.5","type":"CoreDNS"},"etcd":{"local":{"dataDir":"/var/lib/etcd","extraArgs":{"cipher-suites":"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"},"imageRepository":"projects.registry.vmware.com/tkg","imageTag":"v3.4.13_vmware.15"}},"imageRepository":"projects.registry.vmware.com/tkg","scheduler":{"extraArgs":{"tls-cipher-suites":"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"}}},"files":[{"content":"---snip---","encoding":"base64","owner":"root:root","path":"/etc/kubernetes/audit-policy.yaml","permissions":"0600"}],"initConfiguration":{"nodeRegistration":{"criSocket":"/var/run/containerd/containerd.sock","kubeletExtraArgs":{"cloud-provider":"external","tls-cipher-suites":"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"},"name":"{{ ds.meta_data.hostname }}"}},"joinConfiguration":{"nodeRegistration":{"criSocket":"/var/run/containerd/containerd.sock","kubeletExtraArgs":{"cloud-provider":"external","tls-cipher-suites":"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"},"name":"{{ ds.meta_data.hostname }}"}},"preKubeadmCommands":["hostname \"{{ ds.meta_data.hostname }}\"","echo \"::1 ipv6-localhost ipv6-loopback\" \u003e/etc/hosts","echo \"127.0.0.1 localhost\" \u003e\u003e/etc/hosts","echo \"127.0.0.1 {{ ds.meta_data.hostname }}\" \u003e\u003e/etc/hosts","echo \"{{ ds.meta_data.hostname }}\" \u003e/etc/hostname"],"useExperimentalRetryJoin":true,"users":[{"name":"capv","sshAuthorizedKeys":["---snip---"],"sudo":"ALL=(ALL) NOPASSWD:ALL"}]},"replicas":1,"version":"v1.21.2+vmware.1"}}
creationTimestamp: "2021-11-11T07:33:37Z"
finalizers:
- kubeadm.controlplane.cluster.x-k8s.io
generation: 2
labels:
cluster.x-k8s.io/cluster-name: tkg-ssc
name: tkg-ssc-control-plane
namespace: default
ownerReferences:
- apiVersion: cluster.x-k8s.io/v1alpha3
blockOwnerDeletion: true
controller: true
kind: Cluster
name: tkg-ssc
uid: 9bd41852-38df-4d12-bb81-7b2bb35fdfa5
resourceVersion: "5787872"
uid: 1532ce9b-2d7e-45f7-b8ab-2d5bd4fe6b7f
spec:
infrastructureTemplate:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: VSphereMachineTemplate
name: tkg-ssc-control-plane-new
namespace: default
kubeadmConfigSpec:
clusterConfiguration:
apiServer:
extraArgs:
audit-log-maxage: "30"
audit-log-maxbackup: "10"
audit-log-maxsize: "100"
audit-log-path: /var/log/kubernetes/audit.log
audit-policy-file: /etc/kubernetes/audit-policy.yaml
cloud-provider: external
tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
dns:
imageRepository: projects.registry.vmware.com/tkg
imageTag: v1.8.0_vmware.5
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
extraArgs:
cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
imageRepository: projects.registry.vmware.com/tkg
imageTag: v3.4.13_vmware.15
imageRepository: projects.registry.vmware.com/tkg
networking: {}
scheduler:
extraArgs:
tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
files:
- content: ---snip---
encoding: base64
owner: root:root
path: /etc/kubernetes/audit-policy.yaml
permissions: "0600"
initConfiguration:
localAPIEndpoint:
advertiseAddress: ""
bindPort: 0
nodeRegistration:
criSocket: /var/run/containerd/containerd.sock
kubeletExtraArgs:
cloud-provider: external
tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
name: '{{ ds.meta_data.hostname }}'
joinConfiguration:
discovery: {}
nodeRegistration:
criSocket: /var/run/containerd/containerd.sock
kubeletExtraArgs:
cloud-provider: external
tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
name: '{{ ds.meta_data.hostname }}'
preKubeadmCommands:
- hostname "{{ ds.meta_data.hostname }}"
- echo "::1 ipv6-localhost ipv6-loopback" >/etc/hosts
- echo "127.0.0.1 localhost" >>/etc/hosts
- echo "127.0.0.1 {{ ds.meta_data.hostname }}" >>/etc/hosts
- echo "{{ ds.meta_data.hostname }}" >/etc/hostname
useExperimentalRetryJoin: true
users:
- name: capv
sshAuthorizedKeys:
- ssh-rsa ---snip---
sudo: ALL=(ALL) NOPASSWD:ALL
replicas: 1
rolloutStrategy:
rollingUpdate:
maxSurge: 1
type: RollingUpdate
version: v1.21.2+vmware.1
status:
conditions:
- lastTransitionTime: "2021-11-22T14:40:53Z"
status: "True"
type: Ready
- lastTransitionTime: "2021-11-11T07:35:55Z"
status: "True"
type: Available
- lastTransitionTime: "2021-11-11T07:33:39Z"
status: "True"
type: CertificatesAvailable
- lastTransitionTime: "2021-11-22T14:39:32Z"
status: "True"
type: ControlPlaneComponentsHealthy
- lastTransitionTime: "2021-11-22T14:40:53Z"
status: "True"
type: EtcdClusterHealthyCondition
- lastTransitionTime: "2021-11-22T14:40:53Z"
status: "True"
type: MachinesReady
- lastTransitionTime: "2021-11-22T14:39:57Z"
status: "True"
type: MachinesSpecUpToDate
- lastTransitionTime: "2021-11-22T14:40:53Z"
status: "True"
type: Resized
initialized: true
observedGeneration: 2
ready: true
readyReplicas: 1
replicas: 1
selector: cluster.x-k8s.io/cluster-name=tkg-ssc,cluster.x-k8s.io/control-plane
updatedReplicas: 1
Change line 32, to use the new VsphereMachineTemplate called tkg-ssc-control-plane-new. Once you save and quit with :wq! the control plane nodes will be re-deployed.
VMwire 