Github Actions Continuous Integration for a django and Postgresql Web App

GitHub Actions in Action - Setting up Django and Postgres

tl;dr – Here’s a working example of a Django project.

We use GitHub extensively – for client projects, for internal projects & for open source.

It was a matter of time for GitHub to roll their own CI & catch up with Bitbucket Pipelines & GitLab CI.

Having a CI is integral part of our software development process – build & lint on every commit, deploy to staging & production from specific branches.

We use either CircleCI or CodeShip, depending on the project & the needs.

With GitHub Actions now being generally available for everyone, I was itching to give it a go.

The final aim of this article is to provide you with a working Django + Postgres example, share my struggles during the setup.

I’ll go step by step and include some of the errors that you might encounter, while trying to set things up.

Terminology

As with every other CI, I made the mistake to jump right in, start pasting yml configuration around & hoping for the best.

Only after reading some more about GitHub Actions core concepts, I started making progress.

The most helpful page that I found was this – https://help.github.com/en/actions/automating-your-workflow-with-github-actions/workflow-syntax-for-github-actions – I read it from start to end to finally understand the core concepts behind GitHub Actions.

What we need to know:

Workflows

GitHub runs “workflows”, which are yml files, located in .github/workflows directory in your repository. The name of the workflows can be arbitrary. For example, GitHub generates .github/workflows/pythonapp.yml for Python apps.
We can have more than 1 workflow.
Workflows have triggers. You can trigger workflows on “push”, for example. Read more about that here.
1 workflow can have many jobs.

Jobs

One job = one machine of some kind, which runs “steps” for you.
One job groups a bunch of steps together.
Within one workflow, we can have multiple jobs. They run in parallel! That’s very important.
We can create a dependency graph between jobs. For example – “For every successful build on master, deploy to staging on Heroku”. Read more about that here.

Steps

Steps are the commands that we want to execute. Like “run tests” or “install something”. Read more about that here.
Unlike jobs, steps run sequentially, one after the other.
We can reuse steps from elsewhere. For me, this is the most powerful feature of GitHub Actions. This is done quite frequently, so it’s a good idea to read more about that.

Quick summary

Workflows group jobs together.
Jobs group steps together.
Steps are where we put our commands.
Jobs in a workflow run in parallel.
Steps in jobs run sequentially.

The plan

The plan for this article is:

Start with a simple django-admin startproject project.
Create a workflow for it to run tests & migrations via manage.py
Configure Postgres for the app & the CI.
Configure & run tests with py.test

Lets get started!

A simple project

The first thing we do is to create a simple Django project: django-admin startproject github_actions and push that to GitHub.

The only addition that we are going to make is to add requirements.txt next to manage.py and include Django.

By the time of writing, my requirements.txt file looks like that:

Django==3.0.2

We push that & it’s time to setup our first GitHub Actions workflow.

GitHub Actions Workflow for Python

I don’t like yml for configuration. That’s why I’m going to make GitHub generate an initial structure for me.

Go to your GirHub repo & in the Actions tab, select the one for “Python Application”.

Commit the workflow & pull. Now open .github/workflows/pythonapp.yml.

Lets examine the workflow, so we can learn more about it.

The first lines are related to the workflow itself:

name: Python application
on: [push]

That’s the name of the workflow (what you are going to see in GitHub Action’s tab) & the trigger – on every push to every branch.

Next, we see the single job for our workflow:

jobs:
  build:
    runs-on: ubuntu-latest

The name of the job is build – that’s what you are going to see in GitHub Action’s tab.

And since 1 job = 1 machine, we are going to run on latest Ubuntu.

Finally, we have the steps – what we are actually going to execute for this specific job.

steps:
- uses: actions/checkout@v1
- name: Set up Python 3.7
  uses: actions/setup-python@v1
  with:
    python-version: 3.7
- name: Install dependencies
  run: |
    python -m pip install --upgrade pip
    pip install -r requirements.txt
- name: Lint with flake8
  run: |
    pip install flake8
    # stop the build if there are Python syntax errors or undefined names
    flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
    # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
    flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Test with pytest
  run: |
    pip install pytest
    pytest

There’s a bunch of things going on here, so lets first reduce it:

We don’t need flake8 for now, so just remove everything there.
pytest will come after a few steps, so remove that too.

Your workflow should look like that:

name: Python application
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - name: Set up Python 3.7
      uses: actions/setup-python@v1
      with:
        python-version: 3.7
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt

Commit it, go to the Actions tab & watch how it passes. Try to spot where GitHub displays the name of the workflow & the name of the job.

Now, lets drill down into our steps.

Step by step, action by action

The first step we have is - uses: actions/checkout@v1. This means – we are “including” another step, that will get our repository checked out on the machine running the job. This is called an “action” – a reusable unit of code.

As I mentioned previously, actions are quite central to GitHub actions.

This means we can create reusable actions & not paste huge snippets in our yml, which makes it quite hard to maintain.

Again, it’s a good idea to familiarize yourself with the concept of “actions”:

Okay, lets continue with our next step:

- name: Set up Python 3.7
  uses: actions/setup-python@v1
  with:
    python-version: 3.7

This is another GitHub action that we are reusing. The only difference this time is that we have the name key – where we can give a human-readable name to this step, that’s going to be displayed in the GitHub Actions output.

The second one is the with key under uses. This is how we can give “input” or “arguments” to the actions that we are reusing.

We are basically saying – set us up with Python version 3.7. To read more about the setup-python action, you can visit the repository here – https://github.com/actions/setup-python

Okay, lets see our final step:

- name: Install dependencies
  run: |
    python -m pip install --upgrade pip
    pip install -r requirements.txt

No actions are being used here. We have the human-readable name and then we have a new key – run, followed by a |. That’s why I don’t like yml for configuration. Things tend to get cryptic.

This says:

“Run each of the commands, separated by newline, that are at one indent away from the run key”.

Meaning, we will execute the two commands, 1 by 1, installing the needed requirements. Read more about this here.

Migrations & Tests

Now, lets add 2 more steps to run our migrations & tests. After all, we have a Django project!

Here’s how the yml file looks like after adding those 2 steps:

name: Python application
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - name: Set up Python 3.7
      uses: actions/setup-python@v1
      with:
        python-version: 3.7
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
    - name: Run migrations
      run: python manage.py migrate
    - name: Run tests
      run: python manage.py test

Commit & push that, go to Actions tab & observe. Try to understand what’s happening & watch your build succeeds.

Adding Postgres

Now, lets add Postgres to our Django & to GitHub Actions.

Django & Postgres

First, we’ll add Postgres to our Django app. If you haven’t done that before, I suggest reading this wonderful tutorial from DigitalOcean.

We need psycopg2, so we install it – pip install psycopg2 & add it to requirements.txt
Second, we need to update our settings to set the default database to be postgres. Taking this straight from the documentation:

# Database
# https://docs.djangoproject.com/en/3.0/ref/settings/#databases
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'github_actions',
        'USER': 'radorado',
        'PASSWORD': 'radorado',
        'HOST': '127.0.0.1',
        'PORT': '5432',
    }
}

We have a lot of things missing, but lets commit that & see what’ll happen in Actions.

Sadly, the build fails during the Install dependencies step with something like that: Error: pg_config executable not found. – this is an issue coming from psycopg2

psycopg2 dependencies

This library requires some dependencies installed, in order to compile successfully, and to save you several hours of googling around, those are python-dev and libpq-dev. We need to install them not with pip, but with apt-get – Ubuntu’s package manager.

Since every job is running on a machine and we’ve said runs-on: ubuntu-latest, we need to add a step before we install our requirements, to install those dependencies:

name: Python application
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - name: Set up Python 3.7
      uses: actions/setup-python@v1
      with:
        python-version: 3.7
    - name: psycopg2 prerequisites
      run: sudo apt-get install python-dev libpq-dev
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
    - name: Run migrations
      run: python manage.py migrate
    - name: Run tests
      run: python manage.py test

Commit & push & lets see if we can install psycopg2 this time.

Okay – Install dependencies succeeded, but Run migrations failed:

...
django.db.utils.OperationalError: could not connect to server: Connection refused
    Is the server running on host "127.0.0.1" and accepting
    TCP/IP connections on port 5432?

Seems like Postgres is not running on the machine that’s running our job. That’s the next thing we need to fix.

GitHub Actions & Postgres

This is where I spent quite some time. I was looking for an approach like “actions” – plug something in & get it to run.

If you do some googling around & you eventually find this – https://github.com/Harmon758/postgresql-action – a Postgres action! Lets add it & see what’s going to happen:

name: Python application
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - name: Set up Python 3.7
      uses: actions/setup-python@v1
      with:
        python-version: 3.7
    - name: psycopg2 prerequisites
      run: sudo apt-get install python-dev libpq-dev
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
    - uses: harmon758/postgresql-action@v1
      with:
        postgresql version: '11'
    - name: Run migrations
      run: python manage.py migrate
    - name: Run tests
      run: python manage.py test

Sadly, GitHub Actions is getting confused, the step with Postgres is finished, before the database is up and running and we get the following error:

django.db.utils.OperationalError: FATAL: the database system is starting up

Why is this failing? A high level overview of the reason is:

The step runs a Postgres docker image.
Postgres itself needs time to start up.
The next step runs before Postgres has started.
We need some kind of “wait for it” mechanism here.

We can spend more time fighting this (I did), but eventually, you’ll find out about services. The documentation says the following:

Additional containers to host services for a job in a workflow. These are useful for creating databases or cache services like redis. The runner will automatically create a network and manage the life cycle of the service containers.

Sounds exactly what we need.

Again, if you got the google result from above, you probably got that result too – https://github.com/actions/example-services/blob/master/.github/workflows/postgres-service.yml

Now, this is a big yml file and more than one thing is going on there. You can waste a lot of time trying to copy-paste the right thing, especially if you are in a hurry.

Lets declare a service for our job, using the example from above:

name: Python application
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:10.8
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: github_actions
        ports:
          - 5432:5432
        # needed because the postgres container does not provide a healthcheck
        options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
    steps:
    - uses: actions/checkout@v1
    - name: Set up Python 3.7
      uses: actions/setup-python@v1
      with:
        python-version: 3.7
    - name: psycopg2 prerequisites
      run: sudo apt-get install python-dev libpq-dev
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
    - name: Run migrations
      run: python manage.py migrate
    - name: Run tests
      run: python manage.py test

If we run this, python manage.py migrate will fail again. But this time – the error message has changed!

psycopg2.OperationalError: FATAL: password authentication failed for user “radorado”

We managed to connect to postgres inside the build machine & we are now failing to authenticate.

Why? Our database configuration is trying with user radorado and password radorado, while we have provided postgres and postgres as credentials in the service definition.

One important thing to notice is the env key where we pass the user, password & database name for Postgres. This is similar to the with key where we pass input arguments to actions.

Configuring Django for Postgres in GitHub Actions

Now, we can solve the problem from above in many different ways.

The way I’m going to solve it is:

Check if we are running in a workflow inside our settings.py
Change the database settings, if that’s the case

Usually, all CIs export a bunch of environment variables that we can use in our code. This is a nice explanation of environment variables in GitHub Actions.

We are going to check against GITHUB_WORKFLOW:

# Database
# https://docs.djangoproject.com/en/3.0/ref/settings/#databases
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'github_actions',
        'USER': 'radorado',
        'PASSWORD': 'radorado',
        'HOST': '127.0.0.1',
        'PORT': '5432',
    }
}
if os.environ.get('GITHUB_WORKFLOW'):
    DATABASES = {
        'default': {
           'ENGINE': 'django.db.backends.postgresql',
           'NAME': 'github_actions',
           'USER': 'postgres',
           'PASSWORD': 'postgres',
           'HOST': '127.0.0.1',
           'PORT': '5432',
        }
    }

Everything passes ✔️

Adding pytest

First, lets create a new app called website with a model called Page:

from django.db import models
class Page(models.Model):
    name = models.CharField(max_length=255, unique=True)
    slug = models.SlugField(unique=True)

Then, lets create a dummy test:

from django.test import TestCase
from website.models import Page
class WebsiteTests(TestCase):
    def test_page_is_created_successfully(self):
        page = Page(
            name='Home',
            slug='home'
        )
        page.save()

If we just commit that & observe the workflow, the test is going to pass ✔️ (we have python manage.py test)

Now, for pytest, we simply follow the official guide.

If you want, you can:

Separate your requirements file.
Add pytest-django straight to requirements.txt
Just install pytest as a step in the job.

That’s up to you. For the example, I’ll go with the last option.

Here’s our final pythonapp.yml file:

name: Python application
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:10.8
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: github_actions
        ports:
          - 5432:5432
        # needed because the postgres container does not provide a healthcheck
        options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
    steps:
    - uses: actions/checkout@v1
    - name: Set up Python 3.7
      uses: actions/setup-python@v1
      with:
        python-version: 3.7
    - name: psycopg2 prerequisites
      run: sudo apt-get install python-dev libpq-dev
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
        pip install pytest-django
    - name: Run migrations
      run: python manage.py migrate
    - name: Run tests
      run: py.test

Does it spark joy? Yes.

Resources

I hope you learned something.

It’s worthwhile to read some of the documentation of GitHub Actions & understand the underlying mechanism. Blind copy-pasting, as I usually do, leads to a lot of frustration & slow downs.

Here’s a summary of all resources used in that article:

Search This Blog

MUGOYA DIHFAHSIH