CI/CD Pipeline PoC

Host a 3-tier architecture e-commerce microservices application on AKS/EKS/K8S with Azure DevOps based CI/CD pipeline

Repository Setting

    Update .env and specify personal docker repo user name and tag version 
    docker-compose.yaml builds all images and uploads to the repo accordingly
    K8s\helm\values, update the repo to the personal Docker Hub account

Construct CI/CD pipeline

    Build services: specify the Docker Registry Service Connection (docker hub) and the Docker Compose File (./docker-compose.yaml)

    Push services: Same attribute as build services, specify "push service image" in action

    Release:Specify the artifact (github repo), and click on CD (lighting icon) to trigger build when the repo updated. 

    Stages: specify Job (Agent Job) and Tasks (install Helm and helm upgrade)

Cluster configuration

Configure AKS cluster: Node size Standard_D4als_v6 (non-arm) in system node pool
Configure Project setting\Service connections, adding GitHub and AKS Cluster KubeConfig

Source Git
Source Article
My Git Repo 
Docker Hub

CI/CD Project Note (1)

 

Construct EKS based 3 tier architecture project
Robot Shop: A Comprehensive Exploration of its 3-Tier Architecture, 8 Services, and 2 Databases

Create EKS Load Balancer Controller and EBS CSI Driver manually if needed

Reference video

Oracle (OCI) Ampere ARM64v8 Compatible Container

 https://hub.docker.com/r/fredblgr/ubuntu-novnc/tags

https://hub.docker.com/r/dorowu/ubuntu-desktop-lxde-vnc

https://hub.docker.com/_/httpd/tags

K3s Rancher Installation Note

Bulk remove containers/images

docker stop $(docker ps -a -q) 

docker rm $(docker ps -a -q)  

docker rmi $(docker images -a -q) 

Dashboard "404 page not found"

docker run -d --privileged --restart=unless-stopped -p 80:80 -p 443:443

Those are occupied common port, we may use some other port (8081:80,9443:433) to bypass

Reset rancher login password

docker exec -ti <container_id> reset-password

change root password

echo root:imafish | chpasswd

==============

sudo iptables -t nat -A PREROUTING -p tcp --dport 53 -j DNAT --to-destination 127.0.0.53:30001

sudo iptables -t nat -A PREROUTING -p udp --dport 53 -j DNAT --to-destination 127.0.0.53:30001

sudo iptables -D -t nat -A PREROUTING -p udp --dport 53 -j DNAT --to-destination 127.0.0.1:30001

sudo iptables -t nat -v -L PREROUTING -n --line-number

sudo iptables -t nat -D PREROUTING

sudo iptables -I INPUT -p tcp -m tcp --dport 53 -j ACCEPT

sudo iptables -I INPUT -p udp -m udp --dport 53 -j ACCEPT

sudo iptables -S --line-number

Free port 53

https://www.linuxuprising.com/2020/07/ubuntu-how-to-free-up-port-53-used-by.html

Migrate High Volume Data in Google Drive

    When A share a folder to B, the file belong to whoever upload the file and not easy to transfer ownership when A and B are in different organizations it also applies to any org account versus personal Google Drive.

    For moving big files over 20GB to B, B can right click on the file and make a copy, copy to somewhere else, new file will belong to B. For small files, we can use google Colab with rsync command to sync up folder recursively. Each account has daily 750 GB data transfer amount. It generates files with 0 Byte size when transfer quota used up without stopping the rsync. It confuse me since the size of the folders are not identical but rsync stop copying file since file existed.

Useful rsync commends:
recursive rsync, showing progress and not overwrite existed files

!rsync --ignore-existing -ra --progress '/content/drive/MyDrive//FolderA/'

'/content/drive/MyDrive//FolderB'

Find and delete empty files

!find "/content/drive/MyDrive/Backup_Local/SYR/" -size 0  | xargs rm

Find the size for the folder

!du -sh '/content/drive/MyDrive/Backup/' | sort -n -r 

Useful link

https://ourtechroom.com/tech/copy-shared-google-drive-files-folder-to-my-drive/ 

How do Pods communicate in Kubernetes

 https://www.tutorialworks.com/kubernetes-pod-communication/

Connecting Applications with Services
https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/

DevOps Learning Tips

My colleague starting the new role as DevOps engineer, here's the advise from him:

1. Learn to program - write a simple web app. It should have a front end, back end, and a database. I'd recommend using Django or Flask since as a devops engineer you're going to be writing a lot of python/powershell most likely and need to at least be able to read it
   
   
2. OS stuff - set up a local Linux server vm to run your web app, that includes setting up the database and web server yourself. Don't use a GUI
   
   
3. Containers - now that you've written your web app, try to dockerize it. Tip: most databases and web servers you'll encounter can be run as containers. How do you make sure your database container will persist data if you have to re-run it?
   
   
4. CI/CD - setup a Jenkins or GitLab container and configure a CI/CD pipeline for your web app, should deploy as containers. Remember to keep things as immutable as possible, i.e. don't make changes inside running containers
   
   
5. Configuration Management - Use ansible to set up your entire project so far. That is, configure your Linux server to be a Docker host, run Jenkins with your pipeline, and run your containerized web app
   
   
6. Container Orchestration - Kubernetes. Minikube is a single node kubernetes "cluster". Set up Minikube and run your Jenkins container and web app. Bonus points: do all this with Ansible
   
   
7. Infrastructure Provisioning - Use terraform to lift and shift your current setup to AWS. Basic - run containers on EC2 with minikube. Intermediate - use EKS for kubernetes instead of a VM. Advanced - set up your own Kubernetes cluster in AWS to run your Jenkins and web app
   
   
8. Service Mesh - optional or revisit later. Set one up
   
   
9. Monitor - set up monitoring of your k8s with Prometheus. Remember to approach this with IaC and make it immutable as possible
   
   
10. Set up logging with an ELK stack. Remember to approach this with IaC and make it immutable as possible