Continuous Integration for GitHub projects

CI - Continuous Integration - compile code and run all the tests on every push amd ever pull-request.
CD - Continuous Delivery/Deployment.
Run a scheduled job to collect data.
Generate static web sites.
Automatically handle issues (eg. close old issues).

Documentation

GitHub Actions Documentation

Setup

Create directory .github/workflows
Create a YAML file in it.

UI of the GitHub actions

GitHub
Create a repository
There is a link called "Actions"
There are many ready-made workflows one can get started with in a few clicks.

Minimal Ubuntu Linux configuration

name: Minimal Ubuntu

on: push

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Single step
      run: echo Hello World

    - name: Look around
      run: |
        uname -a
        pwd     # /home/runner/work/REPO/REPO
        whoami  # runner
        uptime
        which perl
        which python
        which python3
        which ruby
        which node
        which java
        perl -v
        python -V
        python3 -V
        ruby -v
        node -v
        javac -version
        java -version

Minimal Windows configuration

name: Minimal Windows

on: push

jobs:
  build:
    runs-on: windows-latest
    steps:
    - name: Single step
      run: echo Hello World

    - name: Look around
      run: |
        uname -a
        pwd
        whoami
        uptime
        which perl
        which python
        #which python3
        which ruby
        which node
        which java
        perl -v
        python -V    # 3.7.9
        #python3 -V
        ruby -v
        node -v
        javac -version
        java -version

Minimal MacOS configuration

name: Minimal MacOS

on: push

jobs:
  build:
    runs-on: macos-latest
    steps:
    - name: Single step
      run: echo Hello World

    - name: Look around
      run: |
        uname -a
        pwd
        whoami
        uptime
        which perl
        which python
        which python3
        which ruby
        which node
        which java
        perl -v
        python -V
        python3 -V
        ruby -v
        node -v
        javac -version
        java -version

Name of a workflow

name

name: Free Text defaults to the filename

Triggering jobs

on
triggers
Single event

on: push

Multiple events

on: [push, pull_request]

Run on "push" in every branch.
Run on "pull_request" if it was sent to the "dev" branch.
scheduled every 5 minutes (cron config)

name: Triggers

on:
  push:
    branches: '*'
  pull_request:
    branches: 'dev'
  schedule:
    - cron: '*/5 * * * *'

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Look around
      run: |
        echo $GITHUB_EVENT_NAME
        printenv | sort

Manual events (via POST request)

Environment variables

env:
   DEMO_FIELD: value

Demo Environment variables

Matrix (env vars)

matrix
strategy
fail-fast
matrix

name: Matrix environment variables

on: push

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: true
      matrix:
        fruit:
          - Apple
          - Banana
        meal:
          - Breakfast
          - Lunch
    steps:
    - name: Single step
      env:
         DEMO_FRUIT: ${{ matrix.fruit }}
         DEMO_MEAL:  ${{ matrix.meal }}
      run: |
        echo $DEMO_FRUIT for $DEMO_MEAL

GitHub Action Jobs

Jobs run parallel by default
Demo GitHub Actions in parallel

GitHub Actions - Runners - runs-on

runs-on
runs-on
Linux, Windows, or MacOS running on Azure
GitHub hosted runners

Disable GitHub Action workflow

In the Settings/Actions of your repository you can enable/disable "Actions"

Disable a single GitHub Action job

Adding a if-statement that evaluates to false to the job
See literals

jobs:
  job_name:
    if: ${{ false }}  # disable for now

Disable a single step in a GitHub Action job

Adding an if-statement that evaluates to false to the step:

jobs:
  JOB_NAME:
    # ...
    steps:
    - name: SOME NAME
      if: ${{ false }}

Schedule and conditional runs

name: Push and schedule

on:
  push:
    branches: '*'
  pull_request:
    branches: '*'
  schedule:
    - cron: '*/5 * * * *'

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Single step
      run: |
        echo Hello World
        echo $GITHUB_EVENT_NAME

    - name: Look around
      run: |
        printenv | sort

    - name: Conditional step (push)
      if: ${{ github.event_name == 'push' }}
      run: |
        echo "Only run on a push"

    - name: Conditional step (schedule)
      if: ${{ github.event_name == 'schedule' }}
      run: |
        echo "Only run in schedule"

    - name: Conditional step (pull_request)
      if: ${{ github.event_name == 'pull_request' }}
      run: |
        echo "Only run in pull-request"


    - name: Step after
      run: |
        echo "Always run"

Available GitHub actions

Actions

Create multiline file in GitHub Action

In this workflow example you can see several ways to creta a file from a GitHub Action workflow.

I am not sure if doing so is a good practice or not, I'd probbaly have a file someone in the repository and a script that will copy it, if necessary. Then I'd call that script in my YAML file.

name: Create file

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Create file
      run: |
        printf "Hello\nWorld\n" > hw.txt

    - name: Create file
      run: |
        echo First        > other.txt
        echo Second Line >> other.txt
        echo Third       >> other.txt


    - name: Show file content
      run: |
        pwd
        ls -la
        cat hw.txt
        cat other.txt


    - name: Create directory and create file in homedir
      run: |
        ls -la ~/
        mkdir ~/.zorg
        echo First        > ~/.zorg/home.txt
        echo Second Line >> ~/.zorg/home.txt
        echo Third       >> ~/.zorg/home.txt
        ls -la ~/.zorg/

    - name: Show file content
      run: |
        ls -la ~/
        cat ~/.zorg/home.txt

OS Matrix (Windows, Linux, Mac OSX)

name: OS Matrix

on: push

jobs:
  build:
    strategy:
      fail-fast: false
      matrix:
        runner: [ubuntu-latest, macos-latest, windows-latest]

    runs-on: ${{matrix.runner}}
    steps:
    - uses: actions/checkout@v3

    - name: View environment
      run: |
        uname -a
        printenv | sort

Change directory in GitHub Actions

name: cd

on:
  push:
    branches: '*'

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Experiment
      run: |
        pwd             # /home/runner/work/try/try
        mkdir hello
        cd hello
        pwd             # /home/runner/work/try/try/hello

Install packages on Ubuntu Linux in GitHub Actions

name: Install Linux packages

on:
  push:
    branches: '*'

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Install package
      run: |
        sudo apt-get -y install tree
        which tree

Generate GitHub pages using GitHub Actions

name: Generate web page

on:
  push:
    branches: '*'
  schedule:
    - cron: '*/5 * * * *'
#  page_build:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Environment
      run: |
        printenv | grep GITHUB | sort

    - name: Create page
      run: |
        mkdir -p docs
        date >> docs/dates.txt
        echo '<pre>'            > docs/index.html
        sort -r docs/dates.txt >> docs/index.html
        echo '</pre>'          >> docs/index.html

    - name: Commit new page
      if: github.repository == 'szabgab/try'
      run: |
        GIT_STATUS=$(git status --porcelain)
        echo $GIT_STATUS
        git config --global user.name 'Gabor Szabo'
        git config --global user.email 'szabgab@users.noreply.github.com'
        git add docs/
        if [ "$GIT_STATUS" != "" ]; then git commit -m "Automated Web page generation"; fi
        if [ "$GIT_STATUS" != "" ]; then git push; fi

Workflow Dispatch (manually and via REST call)

name: Push and Workflow Dispatch

on:
  push:
    branches: '*'
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Single step
      run: |
        printenv | grep GITHUB | sort

Run in case of failure

name: Failure?

on:
    push:
        branches: '*'
    pull_request:
        branches: '*'

jobs:
  build:
    runs-on: ubuntu-latest
    name: Job

    steps:
    - uses: actions/checkout@v3

    - name: Step one
      run: |
        echo Always runs
        #ls blabla

    - name: Step two (run on failure)
      if: ${{ failure() }}
      run: echo There was a failure

    - name: Step three
      run: |
          echo Runs if there was no failure
          #ls blabla

You can create a step that will run only if any of the previous steps failed. In this example if you enable the "ls" statement in "Step one" it will fail that will make Step two execute, but Step three won't because there was a failure.

On the other hand if Step One passes then Step Two won't run. Step three will then run.

A failure in step three (e.g. by enabling the ls statement) will not make step Two run.

Setup Droplet for demo

apt-get update
apt-get install -y python3-pip python3-virtualenv
pip3 install flask

copy the webhook file

FLASK_APP=webhook flask run --host 0.0.0.0 --port 80

GitHub Repository used for demo

Integrated feature branches

Commit back (See Generate GitHub Pages)
Don't allow direct commit to "prod"
Every push to a branch called /release/something will check if merging to "prod" would be a fast forward, runs all the tests, merges to "prod" starts deployment.

Deploy using Git commit webhooks

Go to GitHub repository
Settings
Webhooks

Payload URL: https://deploy.hostlocal.com/
Content tpe: application/json
Secret: Your secret

from flask import Flask, request
import logging
import hashlib
import hmac

app = Flask(__name__)
app.logger.setLevel(logging.INFO)

@app.route("/")
def main():
    return "Deployer Demo"

@app.route("/github", methods=["POST"])
def github():
    data_as_json = request.data
    #app.logger.info(request.data_as_json)
    signature = request.environ.get('HTTP_X_HUB_SIGNATURE_256', '')
    app.logger.info(signature)  # sha256=e61920df1d6fb1b30319eca3f5e0d0f826b486a406eb16e46071c6cdd0ce3f9b
    GITHUB_SECRET = 'shibboleth'
    expected_signature = 'sha256=' + hmac.new(GITHUB_SECRET.encode('utf8'), data_as_json, hashlib.sha256).hexdigest()
    app.logger.info(expected_signature)
    if signature != expected_signature:
        app.logger.info('invalid signature')
        return "done"

    app.logger.info('verified')

    data = request.json
    #app.logger.info(data)
    app.logger.info(data['repository']['full_name'])
    app.logger.info(data['after'])
    app.logger.info(data['ref']) # get the branch

    # arbitrary code

    return "ok"

@app.route("/action", methods=["POST"])
def action():
    app.logger.info("action")
    secret = request.form.get('secret')
    #app.logger.info(secret)
    GITHUB_ACTION_SECRET = 'HushHush'
    if secret != GITHUB_ACTION_SECRET:
        app.logger.info('invalid secret provided')
        return "done"

    app.logger.info('verified')
    sha = request.form.get('GITHUB_SHA')
    app.logger.info(sha)
    repository = request.form.get('GITHUB_REPOSITORY')
    app.logger.info(repository)

    # arbitrary code

    return "ok"

Deploy from GitHub Action

Go to GitHub repository
Settings
Environments
New Environment
Name: DEPLOYMENT
Add Secret:
Name: DEPLOY_SECRET
Value: HushHush
curl from GitHub action
we need to send a secret, a repo name, and a sha

Deploy using ssh

ssh-keygen -t rsa -b 4096 -C "user@host" -q -N "" -f ./do
ssh-copy-id -i do.pub user@host

Add Secret:
Name: PRIVATE_KEY
Value: ... the content of the 'do' file ...

ssh-keyscan host

Add Secret:
Name: KNOWN_HOSTS
Value: ... the output of the keyscan command ...

Artifact

In the first job we create a file called date.txt and save it as an artifact.
Then we run 3 parallel jobs on 3 operating systems where we dowload the artifact and show its content.

name: OS and Perl Matrix

on: push

jobs:
  build:
    runs-on: ubuntu-latest
    name: Build
    steps:
      - uses: actions/checkout@v3

      - name: View environment
        run: |
          uname -a
          printenv | sort

      - name: Build
        run: |
          date > date.txt
          cat date.txt

      - name: Archive production artifacts
        uses: actions/upload-artifact@v2
        with:
          name: the-date
          path: |
            date.txt

  test:
    needs: build
    strategy:
      fail-fast: false
      matrix:
        runner: [ubuntu-latest, macos-latest, windows-latest]
    runs-on: ${{matrix.runner}}
    name: OS ${{matrix.runner}}

    steps:
      - name: View environment
        if: ${{ ! startsWith( matrix.runner, 'windows-' )  }}
        run: |
          uname -a
          printenv | sort
          ls -l

      - name: Download a single artifact
        uses: actions/download-artifact@v2
        with:
          name: the-date

      - name: View artifact on Linux and OSX
        if: ${{ ! startsWith( matrix.runner, 'windows-' )  }}
        run: |
          ls -l
          cat date.txt
          date

      - name: View artifact on Windows
        if: ${{ startsWith( matrix.runner, 'windows-' )  }}
        run: |
          dir
          type date.txt
          date

Lock Threads

Automatically lock closed Github Issues and Pull-Requests after a period of inactivity.
lock-threads

name: 'Lock Threads'

on:
  schedule:
    - cron: '0 0 * * *'

jobs:
  lock:
    runs-on: ubuntu-latest
    steps:
      - uses: dessant/lock-threads@v2
        with:
          github-token: ${{ github.token }}
          issue-lock-inactive-days: '14'

GitHub Actions examples

GitHub Workflows

github-workflows - simple reusable workflow
Example:
the workflow in osdc-2023-01-perl uses osdc-site-generator

GitHub Actions for Perl

Goals

Setup CI for CPAN modules.
CI for non-CPAN Perl code.
Collect data and generate web site.

CPAN Testers

e.g. DBI and DBI Matrix

Perl with Makefile.PL

Makefile.PL with perldocker/perl-tester image.

Case studies

Examples - Perl

Perl Tester Docker Image

Perl tester Docker image

CI Perl Tester Helpers

Set of scripts (available via action syntax as well), all of them included in perltester dockers

CI Perl Tester Helpers

GitHub Action to setup perl environment in the marketplace

setup-perl-environment

Perl and OS matrix

name: OS and Perl Matrix

on: push

jobs:
  build:
    strategy:
      fail-fast: false
      matrix:
        runner: [ubuntu-latest, macos-latest, windows-latest]
        perl: [ '5.32', '5.30' ]
    runs-on: ${{matrix.runner}}
    name: OS ${{matrix.runner}} Perl ${{matrix.perl}}
    steps:
    - uses: actions/checkout@v3

    - name: Set up perl
      uses: shogo82148/actions-setup-perl@v1
      with:
          perl-version: ${{ matrix.perl }}
          #distribution: strawberry

    - name: View environment
      run: |
        uname -a
        printenv | sort
        perl -v

    #- name: Install cpanm
    #  if: ${{ matrix.runner != "windows-latest" }}
    #  run: |
    #    curl -L https://cpanmin.us | perl - App::cpanminus

    - name: Install module
      run: |
        cpanm Module::Runtime
    #    cpanm MetaCPAN::Client

Perl and OS matrix - show error logs

name: CI

on:
    push:
        branches: '*'
    pull_request:
        branches: '*'

jobs:
  perl-job:
    strategy:
      fail-fast: false
      matrix:
        runner: [ubuntu-latest, macos-latest, windows-latest]
        perl: [ '5.32', '5.30' ]

    runs-on: ${{matrix.runner}}
    name: OS ${{matrix.runner}} Perl ${{matrix.perl}}

    steps:
    - uses: actions/checkout@v3

    - name: Set up perl
      uses: shogo82148/actions-setup-perl@v1
      with:
          perl-version: ${{ matrix.perl }}
          #distribution: strawberry

    - name: Install dependencies
      run: |
          cpanm --notest Module::Install
          cpanm --installdeps --notest .

    - name: Show content of log files on Linux
      if: ${{ failure() && startsWith( matrix.runner, 'ubuntu-' )  }}
      run: cat /home/runner/.cpanm/work/*/build.log

    - name: Show content of log files on Mac
      if: ${{ failure() && startsWith( matrix.runner, 'macos-' )  }}
      run: cat /Users/runner/.cpanm/work/*/build.log

    - name: Show content of log files on Windows
      if: ${{ failure() && startsWith( matrix.runner, 'windows-' )  }}
      run: cat C:/Users/RUNNER~1/.cpanm/work/*/build.log

    - name: Regular Tests
      run: |
          perl Makefile.PL
          make
          make test

Perl and OS matrix - avoid warnings

name: CI

on: push

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        runner: [ubuntu-latest, macos-latest, windows-latest]
        perl: [ '5.32' ]

    runs-on: ${{matrix.runner}}
    name: OS ${{matrix.runner}} Perl ${{matrix.perl}}

    steps:
    - uses: actions/checkout@v3

    - name: Set up perl
      if: ${{ startsWith( matrix.runner, 'windows-' )  }}
      uses: shogo82148/actions-setup-perl@v1
      with:
          perl-version: ${{ matrix.perl }}
          distribution: ${{ ( startsWith( matrix.runner, 'windows-' ) && 'strawberry' ) || 'default' }}

    - name: Show Perl Version
      run: |
        perl -v

The Perl Planetarium

Perl Planetarium

About Github Action for Perl

Presentation of Olaf Alders
Slides of Olaf Alders

GitHub Actions for Python

Python

A demo to show a simple task before we start learning about the YAML files from scratch

name: Python

on: push

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3

    - name: Setup Python
      uses: actions/setup-python@v2

    - name: Install dependencies
      run: pip install -r requirements.txt

    - name: Check Python version
      run: python -V

    - name: Test with pytest
      run: pytest

pytest


def test_demo():
    assert True

Examples - Python

Python with Matrix

name: Python Matrix

on: push

jobs:
  build_python:

    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.8]

    steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python-version }}

    - name: Install dependencies
      run: pip install -r requirements.txt

    - name: Check Python version
      run: python -V

    - name: Test with pytest
      run: pytest

GitHub Actions for NodeJS

NodeJS and OS matrix

name: CI

on: push

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        runner: [ubuntu-latest, macos-latest, windows-latest]
        nodejs: [ 14.15, 12 ]

    runs-on: ${{matrix.runner}}
    name: OS ${{matrix.runner}} NodeJS ${{matrix.nodejs}}

    steps:
    - uses: actions/checkout@v3

    - name: Use Node.js ${{ matrix.nodejs }}
      uses: actions/setup-node@v1
      with:
        node-version: ${{ matrix.nodejs }}

    - name: Show NodeJS Version
      run: |
        node -v

Keyboard shortcuts

Continuous Integration for GitHub projects