Argo workflow 를 활용하여 CI/CD 구축하기 (SonarQube)

argo workflow 와 설치 방법은 아래를 참고한다.

https://kmaster.tistory.com/25

Argo workflow 를 활용하여 CI/CD 구축하기 (기본설치)

 

이번 시간에는 github에서 받은 소스의 품질을 검사하는 SonarQube 연동 방법을 알아보자. 먼저 SonarQube 에 대해 간략히 알아본 후 argo workflow에서 sonarqube 연동방법을 테스트 해 보겠다.

 

SonarQube

SonarQube는 ( 위키백과 )

소나큐브(SonarQube, 이전 이름: Sonar)는 20개 이상의 프로그래밍 언어에서 버그, 코드 스멜, 보안 취약점을 발견할 목적으로 정적 코드 분석으로 자동 리뷰를 수행하기 위한 지속적인 코드 품질 검사용 오픈 소스 플랫폼이다. 소나소스(SonarSource)가 개발하였다. 소나큐브는 중복 코드, 코딩 표준, 유닛 테스트, 코드 커버리지, 코드 복잡도, 주석, 버그 및 보안 취약점의 보고서를 제공한다.

소나큐브는 매트릭스 역사를 기록하고 점진 그래프를 제공한다. 소나큐브는 완전 자동화된 분석과 메이븐, 앤트, 그래들, MS빌드, 지속적 통합(CI) 도구(밤부, 젠킨스, 허드슨 등)와의 연동을 제공한다.

 

주요 컴포넌트

 

출처 : docs.sonarqube.org

설치

 

# helm repo add sonarqube https://SonarSource.github.io/helm-chart-sonarqube
# helm repo update
# kubectl create namespace sonarqube
# helm upgrade --install -n sonarqube sonarqube sonarqube/sonarqube --set postgresql.persistence.enabled="true" --set postgresql.persistence.storageClass="mystorage"

(storageclass는 환경에 맞게 수정해야 한다.)

 

설치된 내역은 아래와 같다.

 

# kubectl get all -n sonarqube
NAME                         READY   STATUS    RESTARTS   AGE
pod/sonarqube-postgresql-0   1/1     Running   0          90m
pod/sonarqube-sonarqube-0    1/1     Running   0          90m

NAME                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/sonarqube-postgresql            ClusterIP   10.100.195.37    <none>        5432/TCP         90m
service/sonarqube-postgresql-headless   ClusterIP   None             <none>        5432/TCP         90m
service/sonarqube-sonarqube             NodePort    10.110.147.106   <none>        9000:30940/TCP   90m

NAME                                    READY   AGE
statefulset.apps/sonarqube-postgresql   1/1     90m
statefulset.apps/sonarqube-sonarqube    1/1     90m

 

SonarQube Web Console을 위한 Ingress는 다음과 같다.

 

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sonarqube-ingress
  namespace: sonarqube
spec:
  rules:
  - host: "sonarqube.10.60.100.40.nip.io"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: sonarqube-sonarqube
            port:
              number: 9000

 

SonarQube 검사 절차

 

먼저 SonarQube 콘솔에 접속해 보자.

 

 

기본 ( admin/admin ) 로그인 후 패스워드를 변경한 후 Update 버튼을 클릭한다.

 

 

우선 Manually로 접속해서 프로젝트 생성을 진행해 보겠다.

 

 

프로젝트 생성 후 Other CI를 선택한다.

 

 

이제 Project에 접근가능한 Token 을 생성해 보자

 

 

Build는 Other로 선택한다. 그럼 아래와 같이 sonar-scanner를 이용하여 검사를 할 수 있는 실행방법이 가이드된다.

 

 

이제 서버에서 sonar-scanner를 다운로드 받아보자.

 

# wget https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-4.7.0.2747-linux.zip

 

그 다음 테스트할 소스를 github에서 받아보자.

 

# git clone https://github.com/authorjapps/spring-boot-integration-test.git # 이 소스는 아무 소스를 참고한 것이니 개발한 소스를 적용하자.

 

소스를 받았으면 maven으로 package를 실행한 후 sonar-scanner를 실행해보자.

# mvn package

# ./sonar-scanner \
  -Dsonar.projectKey=mytest \
  -Dsonar.sources=./sonar-scanning-examples/sonarqube-scanner-maven/maven-basic \
  -Dsonar.host.url=http://10.60.200.121:30940 \
  -Dsonar.login=ffe273d777c888153ba4afb1964102ded37eb65a \
  -Dsonar.sources=spring-boot-integration-test/src/main/java \
  -Dsonar.java.binaries=spring-boot-integration-test/target/classes

WARN: Property 'sonar.sources' with value './sonar-scanning-examples/sonarqube-scanner-maven/maven-basic' is overridden with value 'spring-boot-integration-test/src/main/java'
INFO: Scanner configuration file: /root/sonarscanner/sonar-scanner-4.7.0.2747-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarScanner 4.7.0.2747
INFO: Java 11.0.14.1 Eclipse Adoptium (64-bit)
INFO: Linux 3.10.0-1127.el7.x86_64 amd64
INFO: User cache: /root/.sonar/cache
INFO: Scanner configuration file: /root/sonarscanner/sonar-scanner-4.7.0.2747-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: Analyzing on SonarQube server 9.3.0
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Load global settings
INFO: Load global settings (done) | time=103ms
INFO: Server id: EA8D9556-AX9PKx4vG4lQnLZcb5HE
INFO: User cache: /root/.sonar/cache
INFO: Load/download plugins
INFO: Load plugins index
INFO: Load plugins index (done) | time=56ms
INFO: Load/download plugins (done) | time=171ms
INFO: Process project properties
INFO: Process project properties (done) | time=9ms
INFO: Execute project builders
INFO: Execute project builders (done) | time=2ms
INFO: Project key: mytest
INFO: Base dir: /root/sonarscanner/sonar-scanner-4.7.0.2747-linux/bin
INFO: Working dir: /root/sonarscanner/sonar-scanner-4.7.0.2747-linux/bin/.scannerwork
INFO: Load project settings for component key: 'mytest'
INFO: Load project settings for component key: 'mytest' (done) | time=28ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=94ms
INFO: Load active rules
INFO: Load active rules (done) | time=2095ms
WARN: SCM provider autodetection failed. Please use "sonar.scm.provider" to define SCM of your project, or disable the SCM Sensor in the project settings.
INFO: Indexing files...
INFO: Project configuration:
INFO: 7 files indexed
INFO: Quality profile for java: Sonar way
INFO: ------------- Run sensors on module mytest
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=29ms
INFO: Sensor JavaSensor [java]
INFO: Configured Java source version (sonar.java.source): none
INFO: JavaClasspath initialization
INFO: JavaClasspath initialization (done) | time=3ms
INFO: JavaTestClasspath initialization
INFO: JavaTestClasspath initialization (done) | time=0ms
INFO: Java "Main" source files AST scan
INFO: 7 source files to be analyzed
INFO: Load project repositories
INFO: Load project repositories (done) | time=23ms
INFO: 7/7 source files have been analyzed
WARN: Dependencies/libraries were not provided for analysis of SOURCE files. The 'sonar.java.libraries' property is empty. Verify your configuration, as you might end up with less precise results.
WARN: Unresolved imports/types have been detected during analysis. Enable DEBUG mode to see them.
WARN: Use of preview features have been detected during analysis. Enable DEBUG mode to see them.
INFO: Java "Main" source files AST scan (done) | time=2223ms
INFO: No "Test" source files to scan.
INFO: No "Generated" source files to scan.
INFO: Sensor JavaSensor [java] (done) | time=2654ms
INFO: Sensor JaCoCo XML Report Importer [jacoco]
INFO: 'sonar.coverage.jacoco.xmlReportPaths' is not defined. Using default locations: target/site/jacoco/jacoco.xml,target/site/jacoco-it/jacoco.xml,build/reports/jacoco/test/jacocoTestReport.xml
INFO: No report imported, no coverage information will be imported by JaCoCo XML Report Importer
INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=2ms
INFO: Sensor CSS Rules [javascript]
INFO: No CSS, PHP, HTML or VueJS files are found in the project. CSS analysis is skipped.
INFO: Sensor CSS Rules [javascript] (done) | time=1ms
INFO: Sensor C# Project Type Information [csharp]
INFO: Sensor C# Project Type Information [csharp] (done) | time=1ms
INFO: Sensor C# Analysis Log [csharp]
INFO: Sensor C# Analysis Log [csharp] (done) | time=18ms
INFO: Sensor C# Properties [csharp]
INFO: Sensor C# Properties [csharp] (done) | time=0ms
INFO: Sensor SurefireSensor [java]
INFO: parsing [/root/sonarscanner/sonar-scanner-4.7.0.2747-linux/bin/target/surefire-reports]
INFO: Sensor SurefireSensor [java] (done) | time=1ms
INFO: Sensor HTML [web]
INFO: Sensor HTML [web] (done) | time=5ms
INFO: Sensor Text Sensor [text]
INFO: 7 source files to be analyzed
INFO: 7/7 source files have been analyzed
INFO: Sensor Text Sensor [text] (done) | time=14ms
INFO: Sensor VB.NET Project Type Information [vbnet]
INFO: Sensor VB.NET Project Type Information [vbnet] (done) | time=1ms
INFO: Sensor VB.NET Analysis Log [vbnet]
INFO: Sensor VB.NET Analysis Log [vbnet] (done) | time=16ms
INFO: Sensor VB.NET Properties [vbnet]
INFO: Sensor VB.NET Properties [vbnet] (done) | time=0ms
INFO: ------------- Run sensors on project
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=10ms
INFO: Sensor Java CPD Block Indexer
INFO: Sensor Java CPD Block Indexer (done) | time=56ms
INFO: SCM Publisher No SCM system was detected. You can use the 'sonar.scm.provider' property to explicitly specify it.
INFO: CPD Executor 2 files had no CPD blocks
INFO: CPD Executor Calculating CPD for 5 files
INFO: CPD Executor CPD calculation finished (done) | time=21ms
INFO: Analysis report generated in 93ms, dir size=145.7 kB
INFO: Analysis report compressed in 33ms, zip size=32.9 kB
INFO: Analysis report uploaded in 29ms
INFO: ANALYSIS SUCCESSFUL, you can browse http://10.60.200.121:30940/dashboard?id=mytest
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://10.60.200.121:30940/api/ce/task?id=AX9Py_tBG4lQnLZcb-dn
INFO: Analysis total time: 7.256 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 8.516s
INFO: Final Memory: 19M/70M
INFO: ------------------------------------------------------------------------

 

이렇게 실행하면 아래와 같이 SonarQube console에서 결과를 조회할 수 있다.

 

 

Failed가 발생한 이유는 Quality Gates Condition을 만족하지 못해서이다. Quality Gates Condition은 아래에서 수정가능하다.

 

 

Condition을 변경 후 다시 scanner를 실행하면 아래와 같이 Pass 됨을 확인 할 수 있다.

 

 

지금 까지 SonarQube 에서 Code 품질을 검사하는 방법에 대해서 테스트 해보았는데, 이제 Argo Workflow에서 Sonar-scanner로 검사하는 task를 실행해 보자.

 

Argo Workflow - SonarQube 연동

 

이번 샘플은 Python code를 대상으로 연동해 보자. 먼저 github에서 소스를 clone하는 예제는 다음과 같다.

 

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: input-artifact-git-
spec:
  serviceAccountName: workflow
  entrypoint: git-clone
  templates:
  - name: git-clone
    inputs:
      artifacts:
      - name: argo-source
        path: /src
        git:
          repo: git://github.com/kmaster8/kubernertes.git
          revision: "main"
    container:
      image: busybox
      command: [sh, -c]
      args: ["ls -al"]
      workingDir: /src

 

소스를 가져온다음 위에서 만든 sonarqube와 연동하는 설정을 보자.

 

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: input-artifact-git-
spec:
  serviceAccountName: workflow
  entrypoint: git-clone
  templates:
  - name: git-clone
    inputs:
      artifacts:
      - name: argo-source
        path: /src
        git:
          repo: git://github.com/kmaster8/kubernertes.git
          revision: "main"
    container:
      image: sonarsource/sonar-scanner-cli
      command: [sonar-scanner]
      workingDir: /src
      args: ["-Dsonar.projectKey=mytest",
             "-Dsonar.sources=.",
             "-Dsonar.host.url=http://sonarqube-sonarqube.sonarqube:9000",
             "-Dsonar.login=6a108b3285e6305a3e6edd53339571cba5d030ad"]

 

실행 로그는 아래와 같다.

 

INFO: 14 source files to be analyzed
INFO: 14/14 source files have been analyzed
INFO: Sensor Text Sensor [text] (done) | time=19ms
INFO: Sensor VB.NET Project Type Information [vbnet]
INFO: Sensor VB.NET Project Type Information [vbnet] (done) | time=2ms
INFO: Sensor VB.NET Analysis Log [vbnet]
INFO: Sensor VB.NET Analysis Log [vbnet] (done) | time=25ms
INFO: Sensor VB.NET Properties [vbnet]
INFO: Sensor VB.NET Properties [vbnet] (done) | time=1ms
INFO: ------------- Run sensors on project
INFO: Sensor Zero Coverage Sensor
...
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 14.194s
INFO: Final Memory: 67M/487M
INFO: ------------------------------------------------------------------------

 

SonarQube 콘솔에서 실행 결과를 조회

 

 

그럼 SonarQube를 통해 소스 검사를 한 후 잠시 Pause 시키는 task를 추가해 보자.

 

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: input-artifact-git-
spec:
  serviceAccountName: workflow
  entrypoint: pipe
  templates:
  - name: pipe
    steps:
    - - name: build
        template: git-clone
    - - name: delay
        template: delay

  - name: git-clone
    inputs:
      artifacts:
      - name: argo-source
        path: /src
        git:
          repo: git://github.com/kmaster8/kubernertes.git
          revision: "main"
    container:
      image: sonarsource/sonar-scanner-cli
      command: [sonar-scanner]
      workingDir: /src
      args: args: ["-Dsonar.projectKey=mytest",
             "-Dsonar.sources=.",
             "-Dsonar.host.url=http://sonarqube-sonarqube.sonarqube:9000",
             "-Dsonar.login=6a108b3285e6305a3e6edd53339571cba5d030ad"]
             
  - name: delay
    suspend: {}

 

 

Resume 버튼을 클릭하면 pipeline이 종료된다.

 

다음시간에는 소스 취약성 검사 후 Kaniko로 Container 이미지를 추가하는 과정을 살펴보겠다.