Avi (NSX Advanced Load Balancer) supports Kubernetes Gateway API. This post shows how to install and use the Gateway API to expose applications using this custom resource definition (CRD).
Gateway API is an open source project managed by the SIG-NETWORK community. It is a collection of resources that model service networking in Kubernetes. These resources –https://gateway-api.sigs.k8s.io/
Service, etc – aim to evolve Kubernetes service networking through expressive, extensible, and role-oriented interfaces that are implemented by many vendors and have broad industry support.
For a quick introduction to the Kubernetes Gateway API, read this link and this link from the Avi documentation.
Why use Gateway API?
You would want to use the Gateway API if you had the following requirements:
- Network segmentation – exposing applications from the same Kubernetes cluster to different network segments
- Shared IP – exposing multiple services that use both TCP and UDP ports on the same IP address
NSX Advanced Load Balancer supports both of these requirements through the use of the Gateway API. The following section describes how this is implemented.
The Gateway API introduces a few new resource types:
GatewayClasses are cluster-scoped resources that act as templates to explicitly define behavior for Gateways derived from them. This is similar in concept to StorageClasses, but for networking data-planes.
Gateways are the deployed instances of GatewayClasses. They are the logical representation of the data-plane which performs routing, which may be in-cluster proxies, hardware LBs, or cloud LBs.
AVI Infra Setting
Aviinfrasetting provides a way to segregate Layer-4/Layer-7 virtual services to have properties based on different underlying infrastructure components, like Service Engine Group, intended VIP Network etc.
A sample Avi Infra Setting is as shown below:
apiVersion: ako.vmware.com/v1alpha1 kind: AviInfraSetting metadata: name: aviinfrasetting-tkg-workload-vip spec: seGroup: name: tkgvsphere-tkgworkload-group10 network: vipNetworks: - networkName: tkg-workload-vip cidr: 172.16.4.64/27 enableRhi: false
Avi Infra Setting is a cluster scoped CRD and can be attached to the intended Services. Avi Infra setting resources can be attached to Services using Gateway APIs.
Gateway APIs provide interfaces to structure Kubernetes service networking.
AKO supports Gateway APIs via the servicesAPI flag in the values.yaml.
The Avi Infra Setting resource can be attached to a Gateway Class object, via the .spec.parametersRef as shown below:
apiVersion: networking.x-k8s.io/v1alpha1 kind: GatewayClass metadata: name: gatewayclass-tkg-workload-vip spec: controller: ako.vmware.com/avi-lb parametersRef: group: ako.vmware.com kind: AviInfraSetting name: aviinfrasetting-tkg-workload-vip
The Gateway object provides a way to configure multiple Services as backends to the Gateway using label matching. The labels are specified as constant key-value pairs, the keys being ako.vmware.com/gateway-namespace and ako.vmware.com/gateway-name. The values corresponding to these keys must match the Gateway namespace and name respectively, for AKO to consider the Gateway valid. In case any one of the label keys are not provided as part of matchLabels OR the namespace/name provided in the label values do no match the actual Gateway namespace/name, AKO will consider the Gateway invalid.
Please see https://avinetworks.com/docs/ako/1.5/gateway/.
kind: Gateway apiVersion: networking.x-k8s.io/v1alpha1 metadata: name: gateway-tkg-workload-vip namespace: default spec: gatewayClassName: gatewayclass-tkg-workload-vip listeners: - protocol: TCP port: 80 routes: selector: matchLabels: ako.vmware.com/gateway-name: gateway-tkg-workload-vip ako.vmware.com/gateway-namespace: default group: v1 kind: Service - protocol: TCP port: 443 routes: selector: matchLabels: ako.vmware.com/gateway-name: gateway-tkg-workload-vip ako.vmware.com/gateway-namespace: default group: v1 kind: Service
How to use GatewayAPI
Tying all of these CRDs together.
A Gateway uses a GatewayClass, which in turn uses an AviInfraSetting. Therefore when a Gateway is used by a Service using the relevant labels, that particular service will be exposed on a network that is referenced by the AviInfraSetting via the .spec.network.vipNetworks
In your helm charts, for any service that needs a LoadBalancer service. You would now want to use ClusterIP instead of LoadBalancer and use Labels such as the following:
apiVersion: v1 kind: Service metadata: name: web-statefulset-service-1 namespace: default labels: ako.vmware.com/gateway-name: gateway-tkg-workload-vip ako.vmware.com/gateway-namespace: default spec: selector: app: nginx ports: - port: 80 targetPort: 80 protocol: TCP type: ClusterIP
ako.vmware.com/gateway-name: gateway-tkg-workload-vip ako.vmware.com/gateway-namespace: default
and the ClusterIP type tells the AKO operator to use the gateways, each gateway is on a separate network segment for traffic separation via the spec.gatewayClassName and conversely the gatewayclass via the spec.parametersRef.name for the AviInfraSetting.