Lab Topology

k8s-multimaster

Getting started to Ingress
ingress

  1. Routing your pod that already exposed by SERVICE to outside world using http/https protocol (URLs).
  2. Ingress can Load Balance traffic and do SSL termination.
  3. You need ingress-controller to use ingress, and load balancer in front of ingress to handle reverse proxy/forwarding.
  4. Ingress-controller option : ingress-nginx, traefik, istio, ingress-haproxy, countour, kong.


Nginx ingress-controller deployment steps:

version info:
- os : ubuntu 20.04
- docker : 20.10.7-0ubuntu1~20.04.1
- kubelet : 1.22.0-00 both of client and server

1. Prepare your k8s cluster.
2. Install ingress-nginx-controller
   k apply -f  https://raw.githubusercontent.com/kubernetes/ingress-nginx/3c0bfc1ca3eb48246b12e77d40bde1162633efae/deploy/static/provider/baremetal/deploy.yaml
   kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission 

   *above was workaround so ingress-controller can running smothly, detail can be found on this link:
   https://github.com/kubernetes/ingress-nginx/issues/7448

3. Makesure all pod/deployment was running successfully.
   ingress-nginx-controller will running for 32177 (http) and 30109 (https)
   screenshot: https://prnt.sc/1qakq8p .

4. For troubleshoot guide: https://kubernetes.github.io/ingress-nginx/troubleshooting/ 
5. Deploy loadbalancer in front of ingress. e.g for http configuration:
   *install nginx latest version first

    user www-data;                                                                                                                               
    worker_processes auto;                                                                                                                       
    pid /run/nginx.pid;                                                                                                                                                                                                                                                                       
    include /usr/share/nginx/modules/*.conf;                                                                                                     

    events {                                                                                                                                     
        worker_connections 1024;                                                                                                                 
    }                                                                                                                                            

    http {                                                                                                                                       
        access_log          /var/log/nginx/access.log;                                                                                           
        error_log           /var/log/nginx/error.log;                                                                                            

        keepalive_timeout   65;                                                                                                                  
        default_type        application/octet-stream;                                                                                            

        include             /etc/nginx/mime.types;                                                                                               
        include             /etc/nginx/conf.d/*.conf;                                                                                            
    }                                                                                                                                            

    stream {                                                                                                                                     
        server {                                                                                                                                 
            listen 80;                                                                                                                           
            proxy_pass backend;                                                                                                                  
        }                                                                                                                                        

        upstream backend {                                                                                                                       
            server 10.1.2.204:32177;                                                                                                             
            server 10.1.2.205:32177;                                                                                                             
        }                                                                                                                                        
    }                         

Minimal working ingress manifest for subdomain access:

1. Create deployment
[email protected]:~/yaml-new#ln -s /usr/bin/kubectl /usr/bin/k
[email protected]:~/yaml-new#k create ns test
[email protected]:~/yaml-new#k apply -f apache.yaml -n test
#apache.apps.kubernetes.id</br>
apiVersion: apps/v1
kind: Deployment
metadata:
  name: apache
  labels:
    app: apache
spec:
  replicas: 3
  selector:
    matchLabels:
      app: apache
  template:
    metadata:
      labels:
        app: apache
    spec:
      containers:
      - name: apache
        image: httpd:latest
        ports:
        - containerPort: 80

[email protected]:~/yaml-new#k apply -f nginx.yaml -n test
#nginx.apps.kubernetes.id</br>
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

2. Expose deployment using ClusterIP.
[email protected]:~/yaml-new#kubectl expose deployment apache --port=80 --protocol=TCP --type=ClusterIP
[email protected]:~/yaml-new#kubectl expose deployment nginx --port=80 --protocol=TCP --type=ClusterIP

3. Makesure service was created for each deployment : k get svc -n test

4. Create ingress using this manifest:
[email protected]:~/yaml-new#k apply -f ingress.yaml
apiVersion: networking.k8s.io/v1                                                    
kind: Ingress                                                                       
metadata:                                                                           
  name: [change-to-deployment-name]                                                                    
  annotations:                                                                      
    kubernetes.io/ingress.class: nginx                                              
spec:                                                                               
  rules:                                                                            
  - host: subdomainx.apps.kubernetes.id                                                 
    http:                                                                           
      paths:                                                                        
      - backend:                                                                    
          service:                                                                  
            name: [change-to-your-service-name]                                                            
            port:                                                                   
              number: [change-to-your-service-port]                                                         
        path: /                                                                     
        pathType: ImplementationSpecific

5. Makesure ingress was created for each service and get ip on ADDRESS Column like below:
[email protected]:~/yaml-new# k get ingress -n test                                                                                                 
NAME      CLASS    HOSTS                        ADDRESS      PORTS   AGE                                                                     
apache    <none>   apache.apps.kubernetes.id    10.1.2.204   80      32m                                                                     
nginx     <none>   nginx.apps.kubernetes.id     10.1.2.204   80      24h

6. Test access from local node while monitoring the log files:
while true; do k describe ingress [INGRESS-NAME] -n test; sleep 10; done
while true; do k logs [INGRESS-CONTROLLER-POD-NAME] -n ingress-nginx; sleep 10; done

curl --header "HOST: apache.apps.kubernetes.id" http://10.1.2.204:31170
curl --header "HOST: nginx.apps.kubernetes.id" http://10.1.2.204:31170

7. Point your domain on your DNS Management or /etc/hosts file, and access it.
8. You can see visitor ip on ingress-controller-pod log files.

Ingress PathType:

  1. ImplementationSpecific
  2. Exact
  3. Prefix
access url via http://cdn.apps.kubernetes.id/bucket

apiVersion: networking.k8s.io/v1                                                    
kind: Ingress                                                                       
metadata:                                                                           
  name: apache                                                                    
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: / #define this                                                              
    kubernetes.io/ingress.class: nginx                                              
spec:                                                                               
  rules:                                                                            
  - host: cdn.apps.kubernetes.id                                                 
    http:                                                                           
      paths:                                                                        
      - backend:                                                                    
          service:                                                                  
            name: apache                                                            
            port:                                                                   
              number: 80                                                        
        path: /cdn                                                                 
        pathType: Prefix #define this

Ingress Wildcard:

access url via http://*.sub.apps.kubernetes.id/login</br>
http://a.sub.apps.kubernetes.id/login</br>
http://b.sub.apps.kubernetes.id/login</br>

apiVersion: networking.k8s.io/v1                                                       
kind: Ingress                                                                          
metadata:                                                                              
  name: apache                                                                         
  annotations:                                                                         
    nginx.ingress.kubernetes.io/rewrite-target: / #define this                         
    kubernetes.io/ingress.class: nginx                                                 
spec:                                                                                  
  rules:                                                                               
  - host: "*.sub.apps.kubernetes.id" #define this                                                
    http:                                                                              
      paths:                                                                           
      - backend:                                                                       
          service:                                                                     
            name: apache                                                               
            port:                                                                      
              number: 80                                                               
        path: /login #define this                                                                   
        pathType: Prefix

Ingress Multiservice:

access url via http://*.sub.apps.kubernetes.id/login
http://*.sub.apps.kubernetes.id/login
http://*.sub.apps.kubernetes.id/products

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: multiservice
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: / #define this
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: "*.sub.apps.kubernetes.id" #define this
    http:
      paths:
      - backend:
          service:
            name: apache
            port:
              number: 80
        path: /login #define this
        pathType: Prefix
      - backend:
          service:
            name: nginx
            port:
              number: 80
        path: /products
        pathType: Prefix

update: ingress-controller deployment issue is fixed on version v1.0.0-beta.3
you dont have to delete webhook anymore
here is how to deploy ingress controller new version:

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.0-beta.3/deploy/static/provider/cloud/deploy.yaml
k apply -f deploy.yaml

#makesure pod/deployment, service, replicateset, and job.batch is running or completed and there is no error on controller
watch -n1 k get all -n ingress-nginx
while true; do k logs -n ingress-nginx deployment.apps/ingress-nginx-controller; sleep 10; done

k describe svc -n ingress-nginx ingress-nginx-controller | grep -i port
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  31822/TCP
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  31805/TCP
HealthCheck NodePort:     31418

configure ip and port in your loadbalancer.

edit your ingressclass and add ingressclass.kubernetes.io/is-default-class: "true" in annotations
so you dont have to define ingressclass in ingress manifest file.

ingressclass

Usefull reference

  1. https://stackoverflow.com/questions/65979766/ingress-with-nginx-controller-not-working-address-missing
  2. https://www.youtube.com/watch?v=dEAtD9PVr_Q
  3. https://awstip.com/nginx-ingress-in-on-premises-kubernetes-873590c2d944
  4. https://kubernetes.io/docs/concepts/services-networking/ingress/

Usefull command

k exec -it nginx-6799fc88d8-2z49v -n test -- /bin/bash