Deploying Fiorano Docker Image to Kubernetes locally using Minikube

Sections below illustrate how to setup and deploy Fiorano API servers in a Kubernetes cluster on a local system using Minikube:

Installing Virtual Box

Download and Install the virtual box from https://www.virtualbox.org/wiki/Downloads

Installing Minikube (for Debian-based Linux)

Install and setup kubectl

$ curl -LO {+}https://storage.googleapis.com/kubernetes-release/release/+$(curl -s {+}https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl+
$ chmod +x ./kubectl
$ sudo mv ./kubectl /usr/local/bin/kubectl  

Install Minikube

$ curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.29.0/minikube-linux-amd64 && chmod +x minikube && sudo cp minikube /usr/local/bin/ && rm minikube  

Start Minikube and check

$ minikube start

$ minikube ssh

$ minikube stop

$ minikube delete

Setting up a Kubernetes cluster

Set up a Network File System for persisting data 

1. Install nfs-server in the host machine and nfs-client in the client machine to access the files.

2. To link a host directory to mount point, edit /etc/fstab and then give the command below:
   

   $ sudo mount -a -t none
   

3. To mount a directory, edit /etc/exports and set the directory.

4. Use the command below to export all directories mentioned in the /etc/exports:

   $ exportfs -ra
   

5. Check if it is mounted successfully using the command below:

   $ showmount -e
   

Start Minikube

• For Windows 10 Professional Edition

  1. Install Chocolatey package manager for ease.

  2. Enable Hyper-V from Windows Features

  3. Using chocolatey, install kubernetes-cli using the command:

     choco install kubernetes-cli
     

  4. Using chocolatey, install minikube using the command:

     choco install minikube
     

  5. To set where the VM needs to be created, add MINIKUBE_HOME=<pathofVMdir> to system properties.

  6. After it creates VM in the path, configure Hyper-V Switch Manager to allow internet access for the VM.

  7. Start minikube vm using the command:

     minikube start --vm-driver=hyperv --cpus=4 --memory=6144 --hyperv-virtual-switch="Primary" -disk-size=40GB
     

• For Debian based Linux

  1. Start minikube and set up resources as per requirement 

     $ minikube start --cpus=4 --memory=6144 --disk-size=40GB
     

  2. Open dashboard using the command:

     $ minikube dashboard
     

Minikube file setup 

1. Ssh into the VM

   $ minikube ssh
   

2. Set user permissions in the data folder in minikube

   $ su -
   $ chmod -R 777 /mnt/sda1/data
   

3. Also note the <VM_IP> of the minikube node

   $ ifconfig
   

Transfer the .tar files saved from docker

The tar file can be copied using scp command

$ cd
$ scp -i .minikube/machines/minikube/id_rsa <tar_file_location> docker@<VM_IP>:/mnt/sda1/data

Example

scp -i .minikube/machines/minikube/id_rsa /home/fiorano/Documents/postgres.tar docker@192.168.99.100:/mnt/sda1/data 

Load the docker images  

$ minikube ssh
$ cd /mnt/sda1/data
$ docker load < <tar_file>

Usage

docker load < fioranoAGSv12_1.tar 

Load the cassandra, postgres, AMS and AGS images

Enable port-forwarding in the minikube  

$ minikube stop

Create a script with contents as below and execute the script to open port forwarding:

Restart the minikube with existing configurations  

$ minikube start
$ minikube dashboard 

Execute the yaml files in the order mentioned below

1. services.yaml file contains the ports required to be exposed by the node for AMS servers and database while also taking care of directing requests. It is of type NodePort and has cluster IP 10.96.0.20
   

   $ kubectl apply -f services.yaml
   

2. ags-services.yaml contains the ports for accessing resource created by a project. It is of type LoadBalancer and has cluster IP 10.96.0.30

   $ kubectl apply -f ags-services.yaml
   

   Ensure that the service name in yaml corresponds to the app name of the deployment or stateful set to be created; otherwise it may not work properly.

   

3. Set the login credentials for postgres comprising of username, password in postgres-config.yaml

    $ kubectl apply -f postgres-config.yaml  
   

4.  Create persistent volumes and their respective claims for Cassandra, Postgres and Fiorano AMS runtimedata.

   $ kubectl apply -f cassandra_pv_pvc.yaml
   $ kubectl apply -f postgres_pv_pvc.yaml
   $ kubectl apply -f fiorano_pv_pvc.yaml
   

   
   

5. Create deployment for multicontainer pods containing Fiorano AMS and databases.

   $ kubectl apply -f fiorano-cassandra-deployments.yaml
   

   

   Wait for 5 minutes to get all the containers running and the workloads to turn green from yellow which means deployments are successful.
   Check if deployment successful by opening the apimgmt dashboard in the browser.
   Common causes of deployment failure:

   1. Fiorano Installer License Expiry

   2. Unable to link to the persistent volume which may be due to unavailability of NFS-server

   3. Ensure that the docker image is loaded in minikube ssh and the docker image name correctly corresponds in the yaml file.

6. Create a stateful set for AGS servers. ( Ensure port forwarding has been enabled in VM before this step)

   $ kubectl apply -f ags-stateful.yaml
   

   
   

Login to the API management dashboard and check for the servers ags_0 etc. available in the server group - serverGroup1.

Deploy projects and check the accessibility of the resource hosted by the gateway servers by changing the IP to localhost:2160  

Scaling

Manual Scaling

Change the number of replicas manually by clicking "Edit" option in stateful sets in dashboard.

Autoscaling

1. For auto-scaling add resource request and limit value in AGS stateful sets.

   resources:
     limits:
       cpu: '4'
       memory: 4G
     requests:
       cpu: '1'
       memory: 2G
   

   

2. Now stop minikube in current configuration 
   

   $ minikube stop
   

3. Start minikube with the following arguments.

   1. For Debian based Linux:

      minikube start --extra-config=controller-manager.horizontal-pod-autoscaler-upscale-delay=1m --extra-config=controller-manager.horizontal-pod-autoscaler-downscale-delay=1m --extra-config=controller-manager.horizontal-pod-autoscaler-sync-period=10s --extra-config=controller-manager.horizontal-pod-autoscaler-downscale-stabilization=1m
      

   2. For Windows: 

      minikube start --vm-driver=hyperv -hyperv-virtual-switch="Primary" --extra-config=controller-manager.horizontal-pod-autoscaler-upscale-delay=1m --extra-config=controller-manager.horizontal-pod-autoscaler-downscale-delay=1m --extra-config=controller-manager.horizontal-pod-autoscaler-sync-period=10s -extra-config=controller-manager.horizontal-pod-autoscaler-downscale-stabilization=1m
      

      

4. Enable the metrics-server add on using the following command:

   minikube addons enable metrics-server  
   

5. Then wait for some time and create the autoscaler

   kubectl autoscale statefulsets ags --cpu-percent=30 --min=1 -max=2
   

   

   For system details:

   kubectl top node
   

   For container details:

   kubectl top pod
   

   For the above-mentioned autoscaler details:

   kubectl describe hpa
   

   

6. For memory-based autoscaling, create a yaml file with the content below and set targetAverageUtilization in it as per requirement:

   apiVersion: autoscaling/v2beta1
   kind: HorizontalPodAutoscaler
   metadata:
    name: fioranoapi12v1mem
    namespace: default
   spec:
    scaleTargetRef:
     apiVersion: apps/v1beta1
     kind: StatefulSet
     name: ags
   minReplicas: 1
   maxReplicas: 2
   metrics:
   - type: Resource
     resource:
      name: memory
      targetAverageUtilization: 40
   

7. To deploy this autoscaler use the command below:

   $ kubectl apply -f <fileName>.yaml 
   

   

8. Use kubectl describe hpa to see what its doing

   

   To delete hpa, use:

   $ kubectl delete hpa fioranoapi12v1mem
   

   This would work provided 2160 has been port forwarded in VM and the ags-service is deployed