Avi Ingress is an alternative to Contour and NGINX ingress controllers.
Tanzu Kubernetes Grid ships with Contour as the default Ingress controller that Tanzu Packages uses to expose Prometheus and Contour. Prometheus and Grafana are configured to use Contour if you set ingress: true in the values.yaml files.
This post details how to set Avi Ingress up and use it to expose these applications using signed TLS certificates.
Let’s start
Install AKO with helm as normal, use ClusterIP in the Avi values.yaml config file.
Reference link to documentation:
https://avinetworks.com/docs/ako/1.9/networking-v1-ingress/
Create secret for ingress certificate, using a wildcard certificate will enable Avi to automatically secure all applications with the TLS certificate.
tls.key and tls.crt in base64 encoded format.
router-certs-default.yaml
apiVersion: v1
kind: Secret
metadata:
name: router-certs-default
namespace: avi-system
type: kubernetes.io/tls
data:
tls.key: --snipped--
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVjVENDQTFtZ0F3SUJBZ0lTQTI0MDJNMStJN01kaTIwRWZlK2hlQitQTUEwR0NTcUdTSWIzRFFFQkN3VUEKTURJeEN6QUpCZ05WQkFZVEFsVlRNUll3RkFZRFZRUUtFdzFNWlhRbmN5QkZibU55ZVhCME1Rc3dDUVlEVlFRRApFd0pTTXpBZUZ3MHlNekF6TWpReE1qSTBNakphRncweU16QTJNakl4TWpJME1qRmFNQ1V4SXpBaEJnTlZCQU1NCkdpb3VkR3RuTFhkdmNtdHNiMkZrTVM1MmJYZHBjbVV1WTI5dE1Ga3dFd1lIS29aSXpqMENBUVlJS29aSXpqMEQKQVFjRFFnQUVmcEs2MUQ5bFkyQUZzdkdwZkhwRlNEYVl1alF0Nk05Z21yYUhrMG5ySHJhTUkrSEs2QXhtMWJyRwpWMHNrd2xDWEtrWlNCbzRUZmFlTDF6bjI1N0M1QktPQ0FsY3dnZ0pUTUE0R0ExVWREd0VCL3dRRUF3SUhnREFkCkJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSEF3SXdEQVlEVlIwVEFRSC9CQUl3QURBZEJnTlYKSFE0RUZnUVVxVjMydlU4Yzl5RFRpY3NVQmJCMFE0MFNsZFl3SHdZRFZSMGpCQmd3Rm9BVUZDNnpGN2RZVnN1dQpVQWxBNWgrdm5Zc1V3c1l3VlFZSUt3WUJCUVVIQVFFRVNUQkhNQ0VHQ0NzR0FRVUZCekFCaGhWb2RIUndPaTh2CmNqTXVieTVzWlc1amNpNXZjbWN3SWdZSUt3WUJCUVVITUFLR0ZtaDBkSEE2THk5eU15NXBMbXhsYm1OeUxtOXkKWnk4d0pRWURWUjBSQkI0d0hJSWFLaTUwYTJjdGQyOXlhMnh2WVdReExuWnRkMmx5WlM1amIyMHdUQVlEVlIwZwpCRVV3UXpBSUJnWm5nUXdCQWdFd053WUxLd1lCQkFHQzN4TUJBUUV3S0RBbUJnZ3JCZ0VGQlFjQ0FSWWFhSFIwCmNEb3ZMMk53Y3k1c1pYUnpaVzVqY25sd2RDNXZjbWN3Z2dFR0Jnb3JCZ0VFQWRaNUFnUUNCSUgzQklIMEFQSUEKZHdCNk1veFUyTGN0dGlEcU9PQlNIdW1FRm5BeUU0Vk5POUlyd1RwWG8xTHJVZ0FBQVljVHlxNTJBQUFFQXdCSQpNRVlDSVFEekZNSklaT3NKMG9GQTV2UVVmNUpZQUlaa3dBMnkxNE92K3ljcTU0ZDZmZ0loQUxOcmNnM0lrNllsCkxlMW1ROHFVZmttNWsxRTZBSDU4OFJhYWZkZlhONTJCQUhjQTZEN1EyajcxQmpVeTUxY292SWxyeVFQVHk5RVIKYSt6cmFlRjNmVzBHdlc0QUFBR0hFOHF1VlFBQUJBTUFTREJHQWlFQW9Wc3ZxbzhaR2o0cmszd1hmL0xlSkNCbApNQkg2UFpBb2UyMVVkbko5aThvQ0lRRGoyS1Q1eWlUOGtRdjFyemxXUWgveHV6VlRpUGtkdlBHL3Zxd3J0SWhjCjJEQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFFczlKSTFwZ3R6T2JyRmd0Vnpsc1FuZC8xMi9QYWQ5WXI2WVMKVE5XM3F1bElhaEZ4UDdVcVRIT0xVSGw0cGdpTThxZ2ZlcmhyTHZXbk1wOUlxQ3JVVElTSnFRblh5bnkyOHA2Zwoyc2NqS2xFSWt2RURvcExoek0ydGpCenc4a1dUYUdYUE8yM0dhcHBHWW14OS9Ma2NkUDVSS0xKMmlRTEJXZlhTCmNQRlNmZWsySEc3dEw1N0s0Uit4eDB4MTdsZ2RLeFdOL1JYQ2RvcHFPY3RyTCtPL0lwWVVWZXNiVzNJbkpFZDkKdjZmS1RmVE84K3JVVnlkajVmUGdFUWJva2Q2L3BDTGdIYS81UVpQMjZ1ZytRa1llUEJvUWRrTkpGOTk4a2NHWQpBZGc0THpJZjdYdU9SNDB4eU90aHIyN1p4Y1FXZnhMM2M4bGJuUlJrMXZNL3pMMDhIdz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0=
k apply -f router-certs-default.yaml
Here is the example online store website deployment using ingress with the certificate. Lets play with this before we get around to exposing Prometheus and Grafana.
sample-ingress.yaml
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: http-ingress-deployment
labels:
app: http-ingress
spec:
replicas: 1
selector:
matchLabels:
app: http-ingress
template:
metadata:
labels:
app: http-ingress
spec:
containers:
- name: http-ingress
image: ianwijaya/hackazon
ports:
- name: http
containerPort: 80
protocol: TCP
imagePullSecrets:
- name: regcred
---
kind: Service
apiVersion: v1
metadata:
name: ingress-svc
labels:
svc: ingress-svc
spec:
ports:
- name: http
port: 80
targetPort: 80
selector:
app: http-ingress
type: ClusterIP
avisvcingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: avisvcingress
annotations:
ako.vmware.com/enable-tls: "true"
labels:
app: avisvcingress
spec:
ingressClassName: avi-lb
rules:
- host: "hackazon.tkg-workload1.vmwire.com"
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: ingress-svc
port:
number: 80
Note that the Service uses ClusterIP and the Ingress is annotated with ako.vmware.com/enable-tls: "true" to use the default tls specified in router-certs-default.yaml. Also add the ingressClassName into the Ingress manifest.
k apply -f sample-ingress.yaml
k apply -f avisvcingress.yaml
k get ingress avisvcingress
NAME CLASS HOSTS ADDRESS PORTS AGE
avisvcingress avi-lb hackazone.tkg-workload1.vmwire.com 172.16.4.69 80 13m
Let’s add another host
Append another host to the avisvcingress.yaml file.
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: avisvcingress
annotations:
ako.vmware.com/enable-tls: "true"
labels:
app: avisvcingress
spec:
ingressClassName: avi-lb
rules:
- host: "hackazon.tkg-workload1.vmwire.com"
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: ingress-svc
port:
number: 80
- host: "nginx.tkg-workload1.vmwire.com"
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: nginx-service
port:
number: 80
k replace -f avisvcingress.yaml
And use the trusty statefulset file to create an nginx webpage. statefulset-topology-aware.yaml
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
namespace: default
labels:
spec:
selector:
app: nginx
ports:
- port: 80
targetPort: 80
protocol: TCP
type: ClusterIP
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
replicas: 3
selector:
matchLabels:
app: nginx
serviceName: nginx-service
template:
metadata:
labels:
app: nginx
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: failure-domain.beta.kubernetes.io/zone
operator: In
values:
- az-1
- az-2
- az-3
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- nginx
topologyKey: failure-domain.beta.kubernetes.io/zone
terminationGracePeriodSeconds: 10
initContainers:
- name: install
image: busybox
command:
- wget
- "-O"
- "/www/index.html"
- https://raw.githubusercontent.com/hugopow/cse/main/index.html
volumeMounts:
- name: www
mountPath: "/www"
containers:
- name: nginx
image: k8s.gcr.io/nginx-slim:0.8
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
- name: logs
mountPath: /logs
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: tanzu-local-ssd
resources:
requests:
storage: 2Gi
- metadata:
name: logs
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: tanzu-local-ssd
resources:
requests:
storage: 1Gi
k apply -f statefulset-topology-aware.yaml
k get ingress avisvcingress
NAME CLASS HOSTS ADDRESS PORTS AGE
avisvcingress avi-lb hackazon.tkg-workload1.vmwire.com,nginx.tkg-workload1.vmwire.com 172.16.4.69 80 7m33s
Notice that another host is added to the same ingress, and both hosts share the same VIP.
Lets add Prometheus to this!
Create a new manifest for Prometheus to use called monitoring-ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: monitoring-ingress
namespace: tanzu-system-monitoring
annotations:
ako.vmware.com/enable-tls: "true"
labels:
app: monitoring-ingress
spec:
ingressClassName: avi-lb
rules:
- host: "prometheus.tkg-workload1.vmwire.com"
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: prometheus-server
port:
number: 80
Note that since Prometheus when deployed by Tanzu Packages is deployed into the namespace tanzu-system-monitoring, we also need to create the new ingress in the same namespace.
Deploy Prometheus following the documentation here, but do not enable ingress in the prometheus-data-values.yaml file, that uses Contour. We don’t want that as we are using Avi Ingress instead.
Add Grafana too!
Create a new manifest for Grafana to use called dashboard-ingress.yaml.
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dashboard-ingress
namespace: tanzu-system-dashboards
annotations:
ako.vmware.com/enable-tls: "true"
labels:
app: dashboard-ingress
spec:
ingressClassName: avi-lb
rules:
- host: "grafana.tkg-workload1.vmwire.com"
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: grafana
port:
number: 80
Note that since Grafana when deployed by Tanzu Packages is deployed into the namespace tanzu-system-dashboards, we also need to create the new ingress in the same namespace.
Deploy Grafana following the documentation here, but do not enable ingress in the grafana-data-values.yaml file, that uses Contour. We don’t want that as we are using Avi Ingress instead.
Summary
Ingress with Avi is really nice, I like it! A single secret to store the TLS certificates and all hosts are automatically configured to use TLS. You also just need to expose TCP 80 as ClusterIP Services and Avi will do the rest for you and expose the application over TCP 443 using the TLS cert.
Here you can see that all four of our applications – hackazon, nginx running across three AZs, Grafana and Prometheus all using Ingress and sharing a single IP address.
Very cool indeed!
k get ingress -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
default avisvcingress avi-lb hackazon.tkg-workload1.vmwire.com,nginx.tkg-workload1.vmwire.com 172.16.4.69 80 58m
tanzu-system-dashboards dashboard-ingress avi-lb grafana.tkg-workload1.vmwire.com 172.16.4.69 80 3m47s
tanzu-system-monitoring monitoring-ingress avi-lb prometheus.tkg-workload1.vmwire.com 172.16.4.69 80 14m
VMwire 