GitHub Actions: End to end CI/CD Pipeline for Django

As promised I am back! I hope you all enjoyed my last blog on GitHub Actions: Basics. Today I will show you how you can take earlier knowledge and make an end to end CI/CD pipeline for a django project using GitHub Actions.

What will we do in here?

What I want to show here is how developers work in their day to day lives. How they work with commits, pushes, pull requests, merges and ci/cd pipelines. For this I will be using a django project. But you all can use any project you like because the basics are all same overall. What might be different is the application specific codes.

• We will open a repo. Create a master and develop branch. Any push to master branch will be deployed to production and develop to the staging.
• All the developers working in the projects should commit their commits to any feature/ branches. For this blog we will be using a branch named feature/feature1
• Generally when the work finishes in the feature/feature1 branch a pull request is submitted to develop branch. On submit request a workflow should run which will check if the formatting of the codes are fine and tests are working alright. You never want to push untested codes to production, right?
• On passing the tests and approval from another reviewer in this case my work account in GitHub @imshetu. I will work on feature branches using my personal account @nashmaniac. When I submit a pull request I will select @imshetu as the reviewer.
• On push to develop branch we will again run the tests and deploy the code to staging server in Google Cloud.
• Later we will submit another pull request from develop to master At this time same code formatting and testing will be checked and approval will be given from a reviewer.
• On push to master we will just run the test and deploy it to production.

Few things to remember, inour case staging and production are the same server. We didn’t want to complicate things by having two servers to maintain. Let’s see this graphically to understand more.

So I will be writing a workflow which will be triggered in pull_request is submitted to or code is push to develop or master branch. Our workflow will contain three jobs

• Health Check Job: For all the testing and formatting check.
• Packaging Job: For docker build and update registry
• Deployment Job: Deploying the published image to cloud.

At this point you might think we have workflows name django and gke in GitHub Actions already, why rewrite it again? I would say if you are comfortable with it, you can definitely do it. In here I will write it step wise so that you can figure out how all these pieces work together.

I know, it will be a bit long. But I assure you, it’s worth the wait.

Application Setup

1. Create a repo in GitHub.
2. Clone the repo in your local. I used ssh key for authentication. If you don’t have one use the https link in clone pop up box.

git clone git@github.com:nashmaniac/github-django-actions.git

3.a. Go to project directory and create a new branch named develop by running git checkout -b develop Now push it to GitHub by running git push -u origin develop

3.b. Here add some branch protection rules to develop and master branch so that no code get pushed to those branch without review.

Left: Adding Branch Protection Rules Right: Branch Protection Rules added.

4. Create another branch named feature/feature1 We will do all our work in here.

5. Initialize a python virtualenv directory, activate it and install django psycopg2-binary pycodestyle gunicorninside the virtualenv and make a requirements.txt by running pip freeze > requirements.txt

6. Create a .gitignore file and added all the necessary files and folders you don’t want in GitHub.

7. Initialize django project running django-admin startproject app .

8. We will create an app in django named users and add some tests there. Create the app using python manage.py startapp users

9. Add a test in users/test.py file like below

users/test.py

10. We need to change out database config in app/settings.py file to make sure our postgres database works. For this delete DATABASES variable in app/settings.py and paste the code below.

app/settings.py

12. We will add a Dockerfile to our project directory which will build our application into an image. We won’t be discussing as it is out of scope. You can find it in the repo.

13. I will be adding a folder named k8s which will contain our helm chart. We will use it during our deployment. For keeping this simple I won’t be discussing it here as it would be out of scope a bit.

14. Create a workflow file name .github/workflows/ci.yaml

Let’s write the workflow!

I know it’s a lot of steps earlier. Let’s write our workflow now. After finishing, we will be creating a service account in Google Cloud for deployments and declare few secret in GitHub repo settings and we will test out the whole shebang.

I will add the whole workflow file here will enough comments to make it understandable. Please go through the comments. It will make it clear.

.github/workflows/ci.yaml

Great, I hope the upper gist makes the workflow clear to us. Few more steps and we are good for testing.

Infrastructure & Secrets Setup

• We need to create a kubernetes cluster in Google Cloud and a service account with Kubernetes Engine Admin Artifact Registry Writer & Storage Admin permission so that it can push and deploy our application. Make sure you download the service account as json in your local.

Left: Kubernetes Create Screen Right: Service Account Creation

• You can setup those with admin permission without any issues because those service accounts credentials won’t be available in logs as those will be set as secrets.
• Go to Settings tab in GitHub Repo and set your secrets.

Left: New Secret Add Right: All secrets set

• Few things to remember. GKE_EMAIL is the email of service account key. You will find it as client_email if you open the json file you downloaded and GKE_PASSWORD would be the content of json file.

Now let’s make a commit in your feature/feature1 branch and push it.

Moment of Truth

I know you all are waiting for this moment. As our code is pushed to feature/feature1 branch. We are all set. I will be adding my work account to this repo so that I can review the pull request.

• Create a pull request and add a reviewer.

Left: Opening a Pull Request with Reviewer Right: Pull Request Approved & Pipeline Passed.

• When I merged the Pull Request after approval from another account in here, A workflow was run and failed due to some helm file issue. So I need to submit another pull request from feature/feature1 to develop and it failed again. So I made some changes and pushed. At the Pull Request #3, it got passed. After merge this is log of push event on develop branch workflow run.

Left: Develop Branch Merged Workflow Run Right: Deployed Workload in Google Cloud (Check the commit id and active revision summary id)

• Let’ check what is deployed actually. I have a Cluster IP Service in kubernetes so I will just port forward it to my local port and check. First connect to kubernetes cluster and run kubectl port-forward svc/app-service 8000:80

Response from Home page after port forwarding

• Let’s submit a pull request from develop to master and approve that.
• Merged to master worked fine but workflow failed due to some nasty helm upgrade issue. But I was glad at this point to show you guys the real scenario. Because in real life troubleshooting the bugs in your code is the most common thing we encounter everyday.

So at last it worked on Pull Request #7. The merged code from develop to master ran successfully.

Left: Pull Request #7 (before Merge) Center: Push Workflow Log Right: Deployment in Google Cloud

What I really want to highlight here is 65d2b9f is the short sha of the master branch commit and the docker image deployed in our cloud is also app:sha-65d2b9f which actually means our built image is deployed successfully.

So finally all worked out. Yay !!!

I was super excited while I was planning to write this article. Yes, faced few impediments during writing this due to helm related issue a bit. But I am glad now that I could finish this. You can find all the codes in here.

Lastly, you can do amazing things with GitHub workflow like opening an issue if workflow fails and assign them to developers, can use CODEOWNERS feature to automatically request review from users and can post slack message too in your slack. This simple automation can make your life lot easier. Hopefully I will write another one on this.

Stay fine and take care of yourself during this COVID lockdown.

Happy Coding !!!