GitLab Pipelines

What is GitLab.org?

• GitLab.org
• Open Source Git hosting platform
• "Complete DevOps platform"
• Core is open source and free
• Various add-ons cost money

What is GitLab.com?

• GitLab.com
• Cloud-based hosting of projects

ALM-Toolbox

• GitLab licenses in Israel
• Tamir Gefen
• ALM-Toolbox

CI - Continuous Integration

• Shorten the feedback cycle
• Run on every push

Documentation

• GitLab.org documentation

• GitLab.com specific documentation

• Documentation of GitLab pipelines

Runner

• An agent (operating system, VM)
• shared runners
• Mostly Linux
• Windows is also avilable
• Mac OSX is also avilable

Docker images

• On top of the runner we use Docker images

.gitlab-ci.yml

• Configuration is in .gitlab-ci.yml
• YAML

Hello World

default:
  script: echo Hello World

• default (required) - an arbitrary name we used for this job

• script: (required) the command to execute in the Docker container

• Current default Docker image is ruby:3.1

• See the pipelines-demo repo.

• See Build / Pipelines menu point.

Hello World in container

default:
  image: busybox:latest
  script: echo Hello World

• default (required) - an arbitrary name we used for this job
• image: the name of the Docker image, by default from Docker Hub
• script: (required) the command to execute in the Docker container

Show what is in the container

default:
   image: busybox:latest
   script:
     - uname -a
     - hostname
     - printenv | sort

Pipeline Hierarchy

• Pipeline Each repository can have a Pipeline (described in the .gitlab-ci.yml file).
• Stages Each Pipeline can have one or more stages. One stage runs after the previous stage finished.
• Jobs Each Stage can have 1 or more jobs. The jobs will run in parallel.
• Script Each job must have a script and can, optionally, have a before_script and an after_script step.

Default Stages

By default there are 3 main stages:

build-job:
    stage: build
    script: echo Build

test-job:
    stage: test
    script: echo Test

deploy-job:
    stage: deploy
    script: echo Deploy

• build
• test
• deploy

Define stages

• If the 3 standard stages are not good for you, you can defined your own stage-names and order them as you like
• Stage names are free text and can includes spaces.

stages:
   - prepare
   - lint
   - compile
   - unittest
   - integration test
   - acceptance test
   - deploy

prepare-job:
    stage: prepare
    script: echo Prepare

lint-job:
    stage: lint
    script: echo Lint

compile-job:
    stage: compile
    script: echo Compile

unit-test-job:
    stage: unittest
    script: echo Unit Test

integration-test-job:
    stage: integration test
    script: echo Integration Test

acceptance-test-job:
    stage: acceptance test
    script: echo Acceptance Test

deploy-job:
    stage: deploy
    script: echo Deploy

Jobs

• Each main-key in the yaml file is a "job".
• The job names are free-text (can contain spaces as well)
• There are a few reserved words such as stages, before_script, after_script, and others

Set stage of job

• Each job is in one of the stages
• Default stage is called test
• It is better to set the stage name explicitely

job-name:
  stage: build

Parallel jobs per stage

build job:
    stage: build
    script: echo Build

unit-test-job:
    stage: test
    script: echo Unit Test

integration-test-job:
    stage: test
    script: echo Integration Test

acceptance-test-job:
    stage: test
    script: echo Acceptance Test

deploy-job:
    stage: deploy
    script: echo Deploy

upload-job:
    stage: deploy
    script: echo  Upload

Manual interaction

• Manual interaction

• Select the gear icon, type in the variable: name and a value. Then let it run.

build:
    stage: build
    script: echo Build

test:
    stage: test
    script: echo Test

deploy:
    stage: deploy
    when: manual
    script: echo Deploy

Manual approval

• Click on the gear as variable type in CODE and as value type in secret.

build:
    stage: build
    script: echo Build

test:
    stage: test
    script: echo Test

deploy:
    stage: deploy
    when: manual
    script:
        - echo Check Deploy
        - echo $CODE
        - if [ "$(echo -n $CODE | sha256sum)" == "2bb80d537b1da3e38bd30361aa855686bde0eacd7162fef6a25fe97bf527a25b  -" ]; then echo ok; else exit 1; fi
        - echo Deploy

To generate such a secret hash use the following command on linux:

echo "'$(echo -n secret | sha256sum)'"

Pick a Windows and MacOS runner

Use Postgres service

• Postgres service on GitLab

Use MongoDB service

Script

• script: it is required and it can be a single command or an array of commands
• before_script and after_script are both optional, but if they exists they must be arrays (even if there is only one element)
• You can have before_script and after_script as a main-key in the YAML file.
• A job that does not have before_script will inherit the central before_script. Same with after_script.

anything:
  image: busybox:latest

  # script is required and can be a single value or an array
  script: echo The script

  # after_script is optional but must be an array
  after_script:
    - echo After script
  # before_script is optional but must be an array
  before_script:
    - echo Before script

Before Script and After Script

before_script:
   - echo Before outside
after_script:
   - echo After outside


my-build:
  stage: build
  script: echo Build script
  after_script:
    - echo After build
  before_script:
    - echo Before build

my-test:
  stage: test
  before_script:
    - echo Before test
  script:
    - echo Test script

my-deploy:
  stage: deploy
  script: echo Deploy script
  after_script:
    - echo After deploy

Variables

• Variables

variables:
   MYCOLOR: Blue

my-test 1:
  stage: test
  variables:
     MYCOLOR: Green
  script:
    - echo color $MYCOLOR

my-test 2:
  stage: test
  script: echo color $MYCOLOR

extends

• extends

variables:
   LOCAL_DIR: 'strawberry'


.code:
   script: |
      echo $LOCAL_DIR
      echo $CASE_NAME

one:
    variables:
       CASE_NAME: Foo
    extends: .code

two:
    variables:
       CASE_NAME: Bar
    extends: .code

Pick a Docker image

job-name:
  image: busybox:latest

• Default image is ruby:2.5 you should really not assume that

Docker registry of GitLab

• Docker Container Registry

image: registry.gitlab.com/ioanrogers/perl-builder:latest

Run on Windows

test_windows:
    stage: test
    tags:
      - shared-windows
    script:
      - dir

Run on Mac OSX

• Currently in closed beta

Interdependence of jobs?

• How can we make one job depende on another one?
• Are stages dependent on each other?
• How to carry the same code from stage to stage?

Cache

try:
    script:
      - echo two
      - ls -l
      - mkdir -p local
      - ls -l local
      - if [ -e local/date.txt ]; then cat local/date.txt; fi
      - date >> local/date.txt

cache:
    paths:
      - local

Python 3 Virtualenv

image: python:3.9
try:
  script:
    - printenv | sort
    - apt-get update
    - apt-get install -y virtualenv
    - virtualenv -p python3 venv
    - source venv/bin/activate
    - pip install flask
    - python --version

keywords:

• rules
• changes

Only run job if specific files changed

# Run if either Dockerfile or requirements.txt file changed
docker:
  rules:
    - changes:
        - Dockerfile
        - requirements.txt
  script:
    - echo docker

keywords:

• artifacts

Artifacts and using them in subsequent jobs

stages:
  - build
  - test

mybuild:
  stage: build
  script:
    - mkdir build
    - date > build/date.txt
    - date > build/other.txt
  artifacts:
    # name: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_NAME"
    paths:
      - build
    expire_in: 1 day

mytest:
  stage: test
  script:
    - find .

GitLab API

• List all the projects of user szabgab using a private access token

curl --silent "https://gitlab.com/api/v4/users/szabgab/projects?private_token=$GITLAB_PRIVATE_TOKEN"
curl --silent --header "PRIVATE-TOKEN: $GITLAB_PRIVATE_TOKEN"  "https://gitlab.com/api/v4/users/szabgab/projects"
# id  (can be also found on the main page of each project it is also available as $CI_PROJECT_ID during the CI run)
# visibility (private or public)

• List of pipelines of project with id 28402932

curl --silent --header "PRIVATE-TOKEN: $GITLAB_PRIVATE_TOKEN"  "https://gitlab.com/api/v4/projects/28402932/pipelines"
# has a field called "status"

• List of jobs:

curl --silent --header "PRIVATE-TOKEN: $GITLAB_PRIVATE_TOKEN"  "https://gitlab.com/api/v4/projects/28402932/jobs"
# has a field called "status"
# has a section called "pipeline"

There is an envrionment variable in the GitLab pipelines called $CI_JOB_TOKEN and it can be used for some things but it is very limited in rights. AFAIK Project level access tokens are only available to paying customer https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html So I am using Personal Access Tokens stored as a secret.

• Go to Settings/CICD/Variables/Add Variable called GITLAB_PRIVATE_TOKEN

GitLab API using Python

• python gitlab

image: python:3.9

always:
  script:
    - pip install python-gitlab
    - python code.py

import os
import gitlab

gl = gitlab.Gitlab('https://gitlab.com/', private_token=os.getenv('GITLAB_PRIVATE_TOKEN'))

# List all the projects
#projects = gl.projects.list()
#for project in projects:
#    print(project)

# Given the current project
project_id = os.getenv('CI_PROJECT_ID')
project = gl.projects.get(project_id)

print("-----------------")
for pipe in project.pipelines.list():
    print(pipe)

print("-----------------")
for job in project.jobs.list():
    print(job)
    print()

GitLab pages

image: python:3.9

stages:
 - build

# Job name must be "pages"
# Content must be in the "public" directory
# It must be makred as artifact
# Then access via https://szabgab.gitlab.io/gl-try-private/
pages:
  stage: build
  script:
    - mkdir public
    - cp src/index.html public/
    - date > public/date.html
    - git rev-parse HEAD > public/sha.html
  artifacts:
    paths:
      - public

<a href="date.html">date</a><br>
<a href="sha.html">sha</a></br>

GitLab pages with Perl

• szabgab.gitlab.io

• repo

• gitlab.szabgab.com

• repo

• CPAN Rocks

Deployment

mkdir ~/.ssh/deploy-demo-gitlab
ssh-keygen -N '' -f ~/.ssh/deploy-demo-gitlab/gitlab_rsa
ssh-keygen -N '' -f ~/.ssh/deploy-demo-gitlab/gitlab_rsa -p
copy the content of gitlab_rsa_pub to ~/.ssh/authorized_keys on the server

Perl projects using GitLab

Find Perl project using GitLab: https://cpan.rocks/ CPAN Digger: https://cpan-digger.perlmaven.com/

Rust projects using GitLab

• Many projects use GitLab.com, many other projects use their own installation of GitLab.

• Rust Digger vcs report

Python projects using GitLab

• PyDigger stats