Jenkins

What are CI/CD and why are they useful?

CI - Continuous Integration
CD - Continuous Deployment / Distribution

Prerequisites for CD

A working CI environment

Prerequisites for CI

Standardized environment.
Command line build system.
Automated Tests: Unit, Integration, Acceptance.

Steps of CI/CD

Triggerd by a new change in the Version Control system
Get the latest source code
Compile the project (if necessary)
Run the unit tests
Save the artifact in a safe storage
Create a package
Set up a test system (might need multiple machines)
Run integration / acceptance tests
Deliver the new version
Deploy the new version
Collect coverage reports
Number of tests - graph

What is Jenkins?

Automation server with lots of plugins
Job management system
Build, test, deploy, etc.

Jenkins Support for Version Control Systems

Git
Mercurial (hg)
Subversion (svn)
Perforce (p4)
ClearCase
Microsoft TFS: Team Foundation Server (Git, TFVC)

Jenkins setup

Central Jenkins server (master)
Jenkins workers (aka. agents)

Install Jenkins

Follow the instructions on the Download Jenkins page.
On your desktop: Windows, OSX, Linux
In a VirtualBox image
On some server in the cloud e.g. Digital Ocean
In Docker

Install Jenkins on Ubuntu

Create Digital Ocean Droplet
Or a Linode
ssh root@
apt-get update
apt-get -y upgrade
wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
echo "deb https://pkg.jenkins.io/debian-stable binary/" >> /etc/apt/sources.list
apt-get update
apt-get install -y jenkins
apt-get install -y python
apt-get install -y virtualenv
apt-get install -y postfix

Download Jenkins

Instructions to install Jenkins
Download and install Jenkins
Download Java JRE for Linux

Run Jenkins war files

java -jar jenkins.war

Install on Ubuntu

apt-get update
apt-get upgrade
apt-get install openjdk-8-jre-headless

apt-get update
apt-get install jenkins

Jenkins modes

Freestyle project
Pipelines
Classic GUI
Blue Ocean (new GUI)

Jenkins: Initial configuration

Browse to http://...:8080/
Install suggested plugins
Add user with password
Manage Jenkins -> Configure System -> System Admin e-mail address

Jenkins report results

Report results, especially failing result by e-mail.
Configure / Post-build Action / E-mail Notification / Recipient: type in an e-mail
Configure Linux box as a mail server (postfix)

Trigger build by commit

Jenkins UI: Jenkins / Manage Jenkins / Configure System
Look for "GitHub Servers" and click on the blue ?-mark it will show the URL:
http://jenkins.szabgab.com:8080/github-webhook/
Remember this
Goto the GitHub page: https://github.com/szabgab/demo-flask-project
Settings
Webhooks
Payload URL: the above URL
Content type: application/x-www-form-urlencoded
Secret: (no secret)
Just the push event
Active
Jenkins UI: Configure the project
Build Triggers
"GitHub hook trigger for GITScm polling"
Save
Make some changes to the project in the local git repo and push it out to Github
Observe that after a few seconds the build starts.

Collect test results: xUnit integration

Use pytest --junitxml=test-results/$BUILD_NUMBER.xml
Configure / Post-build Actions / Publish JUnit test results report / Test report XMLs: test-results/*.xml

No graph error

Look into /var/log/jenkins/jenkins.log

found error about:

WARNING: Error while serving http://jenkins.szabgab.com:8080/job/demo-flask-project/test/trend
...
Caused by: java.awt.AWTError: Assistive Technology not found: org.GNOME.Accessibility.AtkWrapper


Could not initialize class org.jfree.chart.JFreeChart
...
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.jfree.chart.JFreeChart

Solution

sudo vim /etc/java-8-openjdk/accessibility.properties

Comment out the following line:

#assistive_technologies=org.GNOME.Accessibility.AtkWrapper

vim /etc/init.d/jenkins
add `-Djava.awt.headless=true` to the line

$SU -l $JENKINS_USER --shell=/bin/bash -c "$DAEMON $DAEMON_ARGS -- $JAVA $JAVA_ARGS -Djava.awt.headless=true -jar $JENKINS_WAR $JENKINS_ARGS" || return 2

systemctl daemon-reload
service jenkins restart

Deploy

In Jenkins Configure the project again
Add another step to the build process with the following content: ./deploy.sh
Create the deploy.sh in the Git repository with the following content:

#!/bin/bash
echo "Deploy!"

cd /home/gabor/work/demo-flask-project;
/usr/bin/git pull
sudo /usr/sbin/service uwsgi reload

Configure application

visudo

jenkins ALL= NOPASSWD: /usr/sbin/service uwsgi reload

Jenkins master/slave

In Jenkins, as in many other software project the terminology master/slave was in use.
The word master is still in use describing the central Jenkins server.
Instead of the word slave these days the Jenkins project mostly uses worker or agent.
You'll still encounter the word "slave" in many places hence I leave this explanation here.

Jenkins master/agent

Create another VPS and ssh to it
Create user bob: adduser bob
Create ssh public key of the Jenkins user ssh-keygen copy /var/lib/jenkins/.ssh/id_rsa.pub from the Jenkins server to /home/bob/.ssh/authorized_keys on the machine that will be used as an agent and then chown -R bob.bob /home/bob/.ssh/
Verify that user jenkins can ssh to user bob on the agent machine without supplying any password.
If No entry currently exists in the Known Hosts file for this host: Run this as user jenkins on the master:
ssh-keyscan -H 107.170.12.117 >> ~/.ssh/known_hosts
Remote home directory: /home/bob
Launch method: via SSH
SSH Username with private key
Username: bob
Private Key: From the Jenkins master ~/.ssh

Manage Jenkins
    Manage nodes
       New node, Node name: s1   - Permanent Agent

Jenkins resources

Jenkins Freestyle projects

Create new Job

Create new jobs: test-python, Freestyle project
GitHub Porject: https://github.com/szabgab/test-python
Soucre Code Management: Git
     Repository URL: https://github.com/szabgab/test-python.git
     Credentials: None
     Branch Specifier: */master
Build:
     Execute Shell:
     #!/bin/bash
     echo Hello World
Save and click on Build Now, then look at the Console Output

Demo Freestyle project

Freestyle Project

Enter a name: Demo
click on Freestle project
GitHub Project: https://github.com/szabgab/demo-flask-project
(this is only used to create an html link to the project)
Source Code Management
Git: https://github.com/szabgab/demo-flask-project
Save
Build Now
This will clone the current version of the project.
We can see it in the "Workspace"
See "Build history"
Click around, see console output

Scheduling builds by polling the repository

Configure
    Build Triggers
        Poll SCM: H/5 * * * *

Scheduling builds by GitHub hook

Configure
    Build Triggers
        GitHub hook trigger for GITScm polling

Jenkins: Add user - github

Add a user to Jenkins called "github":
Manage Jenkins
   Manage Users
      Create User

Jenkins: GitHub trigger

In GitHub
    Settings
        Integrations and Services
             Add Service: Jenkins (GitHub plugin)
                 http://github:secret@162.243.46.181:8080/github-webhook/

Configure

Build - execute shell

./run_jenkins.sh

Build Now

This will try to run the run_jenkins.sh script that does not exist in our repo and thus it will fail.

Add Jenkins script

In the project directory create the run_jenkins.sh file with the following content:

#!/bin/bash

virtualenv venv
source venv/bin/activate
pip install -r requirements.txt
pytest

Make it executable by running:

chmod +x run_jenkins.sh

Commit it to the git repository and push it out.

Then click on "Build Now" on the Jenkins UI.

It should build successfully now.

Jenkins Pipelines

Pipeline

Docker

Install Docker
apt-get install docker docker.io

docker run hello-world

docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.s

sudo usermod -a -G docker $USER

(for both your own user and for user 'jenkins')

Setup Pipeline

Name: demo-for-pipeline
Multibranch Pipelines
Branch sources
GitHub (no credentials are needed as this is a public project)
Owner: szabgab
Repository: Select: demo-for-pipeline

Groovy in Jenkisfile

First Pipeline

pipeline {
    agent any
    stages {
        stage('build') {
            steps {
                echo 'build'
                sh 'pwd'      // /var/lib/jenkins/workspace/demo-for-pipeline_master-I4VIGTM6JE6TBFWUZBZBVPYDJGBTIK2KHOTD5XDPZN2VMFHSUCCQ
                sh 'id'       // uid=112(jenkins) gid=117(jenkins) groups=117(jenkins),118(docker)
                sh 'uname -a' // Linux s17 4.13.0-43-generic #48-Ubuntu SMP Wed May 16 12:18:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            }
        }
    }
}

Pipeline agents

Multiple stages
Multiple agents

pipeline {
    agent none
    stages {
        stage('build') {
            agent { label 'master' }
            steps {
                echo 'build'
                sh 'pwd'      // /var/lib/jenkins/workspace/demo-for-pipeline_master-I4VIGTM6JE6TBFWUZBZBVPYDJGBTIK2KHOTD5XDPZN2VMFHSUCCQ
                sh 'id'       // uid=112(jenkins) gid=117(jenkins) groups=117(jenkins),118(docker)
                sh 'uname -a' // Linux s17 4.13.0-43-generic #48-Ubuntu SMP Wed May 16 12:18:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            }
        }
        stage('test') {
            agent {
                docker {
                    image 'python'
                }
            }
            steps {
                echo 'test'
                sh 'pwd'      // /var/lib/jenkins/workspace/demo-for-pipeline_master-I4VIGTM6JE6TBFWUZBZBVPYDJGBTIK2KHOTD5XDPZN2VMFHSUCCQ
                sh 'id'       // uid=112 gid=117 groups=117
                sh 'uname -a' // Linux 8a88f60d26c1 4.13.0-43-generic #48-Ubuntu SMP Wed May 16 12:18:48 UTC 2018 x86_64 GNU/Linux
            }
        }
    }
}

agent any
agent none
agent { label 'master' }
agent { docker }

Pipeline post stages

Jenkinsfile

post

Pipeline for our project

Jenkinsfile

artifacts

The Pipline uses git clean -fdx to clean the workspace before running any of our commands so we should not leave any of our files (e.g. junit xml files, artifacts, etc. in the workspace)

Blue Ocean

It is just a plugin...

Jenkins and Python

Jenkins: Testing Python

Setup testing a simple project in Python
Replace the "Execute Shell" by ./run_jenkins

Don't forget to add #!/bin/bash at the begining Virtualenv (clean it every time we use it) Before running the tests make sure the environment is clean git status is clean git status "nothing to commit, working tree clean" git clean -xfd there are no untracked files, not even ones that are ignored. - remove all untracked file

Integrate pylint reporting

Install the "Violations" plugin
Add the Violations to the "Post-build Actions" of the project
In the pylint line put: pylint.log
In "Source Path Pattern" put **/
pip install pylint
pylint --generate-rcfile > pylint.cfg
Update the run_jenkins file to run pylint

Python project to try Jenkins

test-python

Collect test results: xUnit integration

Use pytest --junitxml=test-results/$BUILD_NUMBER.xml
Configure / Post-build Actions / Publish JUnit test results report / Test report XMLs: test-results/*.xml

Jenkins: Private GitHub repository

Create private/public keypair: su - jenkins; ssh-keygen; ENTER * 3
cat .ssh/id_rsa.pub and copy paste it the GitHub repo Settings / Deploy keys
Try cloning the private repo still as use jenkins: git clone git@github.com:szabgab/python-test-private.git
In the Jenkins GUI setup separate job called python-test-private where the Git / Repositories is git@github.com:szabgab/python-test-private.git
Add Jenkins credentials: "SSH Username with private key", Username: git, Private Key: From the Jenkins master ~/.ssh

Jenkins administration

Jenkins Files

/etc/default/jenkins
/usr/share/jenkins
/var/lib/jenkins
/var/lib/jenkins/config.xml

Jenkins: Manual Backup

/var/lib/jenkins/config.xml
/var/lib/jenkins/jobs/
/var/lib/jenkins/users/
logging: tail -f /var/log/jenkins/jenkins.log

Backup Jenkins

Backup Jenkins home $JENKINS_HOME /var/lib/jenkins by default
thinBackup can be used

Configure Matrix Based Authentication

Manage Jenkins
   Configure Global Security
      Authorization: (default: Logged-in users can do anything)
         We set:  Matrix-based security
      !!!! add your user to the table with full rights !!!!
     User github: Job: Read+Build rights

Jenkins configuration files

How can we keep Jenkins itself in some version control?

Keyboard shortcuts

Jenkins