Final Challenge

Table of Contents

1 - Kanban

TODO Doing Done
    Learn about Architecture Diagrams
    Design Processes Diagram
    Design Information Flow
    Clone Repository
    Install docker, git and go
    Learn about Go toolchain
    Play with code from repository
    Run code and tests on docker
    Fix Docker image
    Docker: Jenkins and BlueOcean
    Simple BlueOcean pipeline
    Pipeline: Build step
    Pipeline: Testing step
    Custom docker image with Go
    Jenkins Webhook
    Learn about Sonar Qube
    Install sonarqube
    Integrate Sonar Qube
    Jenkins for Docker Orchestration
    Everything in Docker !!
    Linting and formatting
    CV
    Secure Jenkins
    Fix the code
    JUnit test reporting for Jenkins
    Publishing to github

2 - Architecture Diagram

Link to architecture diagram in Diagrams.net

diagram.png

3 - Process Flow

Link to process flow in Diagrams.net

process-flow.png

4 - Repository

https://github.com/albertoefguerrero/DOTT

5 Learn about go toolchain

  • Setting up GO environment variables (should be done with root)
export GOPATH=~/go
export GOBIN=$GOPATH/bin
export PATH=$PATH:$GOBIN

6 Play with code from repository

  • Proper goop repository is ~github.com/karmakaze/goop. Should change in Dockerfile
go get github.com/karmakaze/goop # should be run as root
goop install
goop go run api.go convert.go
  • Ash is the alpine shell
  • The port is defined in main() in line 66 of file api.go, this should be a variable, not hardcoded
log.Fatal(http.ListenAndServe(":8000", router))
  • Test are run like this
go get github.com/stretchr/testify/assert
goop go test

7 Jenkins with Go

https://plugins.jenkins.io/golang/

8 Installing and running Docker on CentOS 7

sudo yum -y update
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install docker-ce docker-ce-cli containerd.io
sudo systemctl start docker
# sudo docker run hello-world # only to test

9 Building Dockerfile and running

docker build -t gom .
docker run -ti -p 8000:8000 gom

exec commands 

docker exec -it <container> <command>

10 Fixed Dockerfile

Testify required Go 1.13+ so using a newer image of golang Mux required GCC so it was added too 

FROM golang:1.15-alpine

ENV GOPATH=/go:/app
RUN apk add --update git
RUN apk add build-base
WORKDIR /app

COPY Goopfile Goopfile
RUN go get github.com/karmakaze/goop \
    && go get github.com/gorilla/mux \
    && go get github.com/stretchr/testify/assert \
    && goop install

COPY . .
CMD ["ash", "-c", "goop go run api.go convert.go"]
# docker build -t gom .
# docker run -ti -p 8000:8000 gom

11 Jenkins with BlueOcean

It is really easy to create a Pipeline with Jenkins and BlueOcean

11.1 Installing with docker

We can install BlueOcean and jenkins with Docker 

docker run -p 8080:8080 jenkinsci/blueocean

-The documentation for the blue ocean docker

11.2 Webhook

http://3.129.89.220:8081/multibranch-webhook-trigger/invoke?token=token-test

12 DO NOT USE. Creating a custom Dockerfile with Jenkins, BlueOcean, Go and dependencies

FROM jenkinsci/blueocean
USER root
RUN apk add --no-cache git make musl-dev go

# Configure Go
ENV GOROOT /usr/lib/go
ENV GOPATH /go
ENV PATH /go/bin:$PATH
ENV GOBIN $GOPATH/bin

RUN mkdir -p ${GOPATH}/src ${GOPATH}/bin

RUN go get github.com/karmakaze/goop \
    && go get github.com/gorilla/mux \
    && go get github.com/stretchr/testify/assert \
    && go get github.com/pkg/errors

With inspiration from here and here and here

13 Custom Dockerfile with Jenkins BlueOcean and Docker

FROM jenkinsci/blueocean
USER root
RUN apk add --no-cache git make musl-dev
RUN apk add --update docker openrc
RUN rc-update add docker boot
RUN addgroup username docker
RUN apk add docker-compose

docker run -it -v "/var/run/docker.sock:/var/run/docker.sock:rw" image-name /bin/sh

https://stackoverflow.com/questions/54099218/how-can-i-install-docker-inside-an-alpine-container https://wiki.alpinelinux.org/wiki/Docker#Installation https://stackoverflow.com/questions/28302178/how-can-i-add-a-volume-to-an-existing-docker-container

13.1 Running Custom Docker image

  • To create the image we do (Make sure the Dockerfile is in path)
docker build -t custom .
  • To run the image we do
docker run -p 8000:8000 -p 8080:8080 custom

export JAVAHOME=/opt/java/current && export PATH=$PATH:$JAVAHOME/bin &&

export JAVAHOME=/usr/lib/jvm/java-11-openjdk && export PATH="\(JAVA_HOME/bin:\){PATH}" && docker run –network=host -e SONARHOSTURL='http://albertoefg1c.mylabserver.com:9090' –user="\((id -u):\)(id -g)" -v "$PWD:/usr/src" sonarsource/sonar-scanner-cli

http://pietervogelaar.nl/jenkinsfile-docker-pipeline-multi-stage

https://www.kabisa.nl/tech/running-multiple-docker-containers-in-parallel-with-jenkins/

https://stackoverflow.com/questions/49782267/running-multiple-docker-containers-from-a-single-jenkinsfile

13.2 Prerequisites and Overview SonarQube https://docs.sonarqube.org/latest/requirements/requirements/

Hardware Requirements A small-scale (individual or small team) instance of the SonarQube server requires at least 2GB of RAM to run efficiently and 1GB of free RAM for the OS. If you are installing an instance for a large teams or Enterprise, please consider the additional recommendations below. The amount of disk space you need will depend on how much code you analyze with SonarQube. SonarQube must be installed on hard drives that have excellent read & write performance. Most importantly, the "data" folder houses the Elasticsearch indices on which a huge amount of I/O will be done when the server is up and running. Great read & write hard drive performance will therefore have a great impact on the overall SonarQube server performance. SonarQube does not support 32-bit systems on the server side. SonarQube does, however, support 32-bit systems on the scanner side.

14 Sonar Qube

docker run -d --name sonarqube \
    -p 9000:9000 \
    -e SONAR_JDBC_URL=... \
    -e SONAR_JDBC_USERNAME=... \
    -e SONAR_JDBC_PASSWORD=... \
    -v sonarqube_data:/opt/sonarqube/data \
    -v sonarqube_extensions:/opt/sonarqube/extensions \
    -v sonarqube_logs:/opt/sonarqube/logs \
    sonarqube

docker run -d --name sonarqube3 -p 9090:9000 -p 9092:9092 -v sonarqube-conf:/opt/sonarqube/conf -v sonarqube-data:/opt/sonarqube/data -v sonarqube-logs:/opt/sonarqube/logs -v sonarqube-extensions:/opt/sonarqube/extensions sonarqube

14.1 Sonar Scanner

docker run --rm --network host \
       -e SONAR_HOST_URL="http://3.138.247.153/" \
       -e SONAR_LOGIN="99a7536d3c88fc79e7f1dd189f99b4cf59926cc6" \
       -v "/var/jenkins_home/workspace/DOTT_master/cidr_convert_api/go/" \
       sonarsource/sonar-scanner-cli -X

--network host -e SONAR_HOST_URL="http://3.138.247.153/" -e SONAR_LOGIN="99a7536d3c88fc79e7f1dd189f99b4cf59926cc6" -v "/var/jenkins_home/workspace/DOTT_master/cidr_convert_api/go/" 

sonar-scanner \
  -Dsonar.projectKey=test-key1 \
  -Dsonar.sources=. \
  -Dsonar.host.url=http://3.22.117.110 \
  -Dsonar.login=test
  -Dsonar.go.coverage.reportPaths=$WORKSPACE/coverage.out

15 Share directories across docker images

https://www.jenkins.io/doc/book/pipeline/docker/#caching-data-for-containers when using the pipeline we should have 

args '-v $HOME/.m2:/root/.m2'

Notes: First directory is the host, second directory is inside docker If directories don't exists, jenkins will create them

https://docs.docker.com/storage/volumes/#choose-the--v-or---mount-flag

16 Fixing Code

17 Securing jenkins

https://github.com/jenkinsci/credentials-binding-plugin https://fedidat.com/270-jenkinsfile-scripted-secret-text/ https://www.jenkins.io/doc/pipeline/tour/environment/

Environment variables http://albertoefg2c.mylabserver.com:8080/env-vars.html/

18 Coverage

https://docs.sonarqube.org/latest/analysis/languages/go/ https://docs.sonarqube.org/latest/analysis/coverage/ https://community.sonarsource.com/t/golang-code-coverage-difficulties/21348 https://stackoverflow.com/questions/10516662/how-to-measure-code-coverage-in-golang https://community.sonarsource.com/t/coverage-test-data-generate-reports-for-apex-c-c-objective-c-go-js-ts-and-python/9687

19 Junit testing for Jenkins

Jenkins has support for JUnit-style XML reports https://www.jenkins.io/doc/pipeline/tour/tests-and-artifacts/ Golang repository to create them https://github.com/jstemmer/go-junit-report 

# install
go get -u github.com/jstemmer/go-junit-report
# run
go test -v 2>&1 | go-junit-report > report.xml
# run with benchmark
go test -v -bench . -count 5 2>&1 | go-junit-report > report.xml

After that we just create an step in jenkins to upload them

20 Publishing

Package should include both api and go.sum files https://blog.golang.org/using-go-modules

The go command uses the go.sum file to ensure that future downloads of these modules retrieve the same bits as the first download, to ensure the modules your project depends on do not change unexpectedly, whether for malicious, accidental, or other reasons. Both go.mod and go.sum should be checked into version control.

github-release https://github.com/github-release/github-release Upload build artifact to Github as release in Jenkins https://stackoverflow.com/questions/24585609/upload-build-artifact-to-github-as-release-in-jenkins

GO build should be statically linked

This is for linux 

$ env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o app

https://oddcode.daveamit.com/2018/08/16/statically-compile-golang-binary/ https://www.arp242.net/static-go.html

20.1 NO longer possible to run binary only

https://golang.org/pkg/go/build/

"go build" and other commands no longer support binary-only-packages. Import and ImportDir will still set the BinaryOnly flag in packages containing these comments for use in tools and error messages.

21 - Notes

Date: 2020-12-12 sáb 00:00

Author: Alberto Flores

Created: 2020-12-18 vie 14:18

Validate