Using LoadBalancers, Gateways, GatewayClasses, AviInfraSettings, IngressClasses and Ingresses
Gateway API replaces services of type LoadBalancer in applications that require shared IP with multiple services and network segmentation. The Gateway API can be used to meet the following requirements:
- Shared IP – supporting multiple services, protocols and ports on the same load balancer external IP address
- Network segmentation – supporting multiple networks, e.g., oam, signaling and traffic on the same load balancer
NSX Advanced Load Balancer (Avi) supports both of these requirements through the use of the Gateway API. The following section describes how this is implemented.
The Gateway API introduces a few new resource types:
- GatewayClasses are cluster-scoped resources that act as templates to explicitly define behavior for Gateways derived from them. This is similar in concept to StorageClasses, but for networking data-planes.
- Gateways are the deployed instances of GatewayClasses. They are the logical representation of the data-plane which performs routing, which may be in-cluster proxies, hardware LBs, or cloud LBs.
Aviinfrasetting
Avi Infra Setting provides a way to segregate Layer-4/Layer-7 virtual services to have properties based on different underlying infrastructure components, like Service Engine Group, intended VIP Network etc.
A sample Avi Infra Setting is as shown below:
apiVersion: ako.vmware.com/v1alpha1
kind: AviInfraSetting
metadata:
name: aviinfrasetting-tkg-wkld-oam
spec:
seGroup:
name: tkgvsphere-tkgworkload-group10
network:
vipNetworks:
- networkName: tkg-wkld-oam-vip
cidr: 10.223.63.0/26
enableRhi: false
Avi Infra Setting is a cluster scoped CRD and can be attached to the intended Services. Avi Infra setting resources can be attached to Services using Gateway APIs.
GatewayClass
Gateway APIs provide interfaces to structure Kubernetes service networking.
AKO supports Gateway APIs via the servicesAPI flag in the values.yaml.
The Avi Infra Setting resource can be attached to a Gateway Class object, via the .spec.parametersRef as shown below:
apiVersion: networking.x-k8s.io/v1alpha1
kind: GatewayClass
metadata:
name: avigatewayclass-tkg-wkld-oam
spec:
controller: ako.vmware.com/avi-lb
parametersRef:
group: ako.vmware.com
kind: AviInfraSetting
name: aviinfrasetting-tkg-wkld-oam
Gateway
The Gateway object provides a way to configure multiple Services as backends to the Gateway using label matching. The labels are specified as constant key-value pairs, the keys being ako.vmware.com/gateway-namespace and ako.vmware.com/gateway-name. The values corresponding to these keys must match the Gateway namespace and name respectively, for AKO to consider the Gateway valid. In case any one of the label keys are not provided as part of matchLabels OR the namespace/name provided in the label values do no match the actual Gateway namespace/name, AKO will consider the Gateway invalid. Please see https://avinetworks.com/docs/ako/1.5/gateway/.
kind: Gateway
apiVersion: networking.x-k8s.io/v1alpha1
metadata:
name: app-gateway-admin-0
namespace: default
spec:
gatewayClassName: avigatewayclass-tkg-wkld-oam
listeners:
- protocol: UDP
port: 161
routes:
selector:
matchLabels:
ako.vmware.com/gateway-name: app-gateway-admin-0
ako.vmware.com/gateway-namespace: default
group: v1
kind: Service
- protocol: TCP
port: 80
routes:
selector:
matchLabels:
ako.vmware.com/gateway-name: app-gateway-admin-0
ako.vmware.com/gateway-namespace: default
group: v1
kind: Service
- protocol: TCP
port: 443
routes:
selector:
matchLabels:
ako.vmware.com/gateway-name: app-gateway-admin-0
ako.vmware.com/gateway-namespace: default
group: v1
kind: Service
How to use the GatewayAPI
In your helm charts, for any service that needs a LoadBalancer service. You would now want to use ClusterIP instead but use Labels such as the following:
apiVersion: v1
kind: Service
metadata:
name: web-statefulset-service-oam
namespace: default
labels:
ako.vmware.com/gateway-name: app-gateway-admin-0
ako.vmware.com/gateway-namespace: default
spec:
selector:
app: nginx
ports:
- port: 8443
targetPort: 443
protocol: TCP
type: ClusterIP
The Gateway Labels
ako.vmware.com/gateway-name: app-gateway-admin-0
ako.vmware.com/gateway-namespace: default
and the ClusterIP type tells the Avi Kubernetes Operator (AKO) to use the gateways, each gateway is on a separate network segment for traffic separation.
The gateways also have the relevant ports that the application uses, configure your gateway and change your helm chart to use the gateway objects.
Ingress Class
Avi Infra Settings can be applied to Ingress resources, using the IngressClass construct. IngressClass provides a way to configure Controller-specific load balancing parameters and applies these configurations to a set of Ingress objects. AKO supports listening to IngressClass resources in Kubernetes version 1.19+. The Avi Infra Setting reference can be provided in the Ingress Class as shown below:
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
name: avi-ingress-class-oam
spec:
controller: ako.vmware.com/avi-lb
parameters:
apiGroup: ako.vmware.com
kind: AviInfraSetting
name: aviinfrasetting-tkg-wkld-oam
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
name: avi-ingress-class-trf
spec:
controller: ako.vmware.com/avi-lb
parameters:
apiGroup: ako.vmware.com
kind: AviInfraSetting
name: aviinfrasetting-tkg-wkld-trf
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
name: avi-ingress-class-trf
spec:
controller: ako.vmware.com/avi-lb
parameters:
apiGroup: ako.vmware.com
kind: AviInfraSetting
name: aviinfrasetting-tkg-wkld-sigtran
Using IngresClass
The Avi Infra Setting resource can be attached to a Gateway Class object and Ingress Class object, via the .spec.parametersRef. However, using annotations with LoadBalancer object instead of using labels with Gateway API object, you will not be able to use shared protocol and ports on the same IP address. For example, TCP AND UDP 53 on the same LoadBalancer IP address. This is not supported yet, until MixedProtocolLB is supported by Kubernetes.
To provide a Controller to implement a given ingress, in addition to creating the IngressClass object, the ingressClassName should be specified, that matches the IngressClass name. The ingress looks as shown below:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: my-ingress
spec:
ingressClassName: avi-ingress-class-oam
rules:
- host: my-website.my-domain.com
http:
paths:
- path: /foo
backend:
serviceName: web-service-1
servicePort: 443
Using Annotation with Services of type LoadBalancer
Services of Type LoadBalancer can specify the Avi Infra Setting using an annotation as shown below without using Gateway API objects:
annotations:
aviinfrasetting.ako.vmware.com/name: "aviinfrasetting-tkg-wkld-sigtran"
annotations:
aviinfrasetting.ako.vmware.com/name: "aviinfrasetting-tkg-wkld-trf”
annotations:
aviinfrasetting.ako.vmware.com/name: "aviinfrasetting-tkg-wkld-oam"
VMwire 