Kubernetes scan 도구 - popeye

 

 

Popeye는 Kubernetes 클러스터를 스캔하고 배포된 리소스 및 구성과 관련된 잠재적인 문제를 보고하는 유틸리티이다.

또한 Kubernetes Cluster에 Metric-Server를 사용하고 있다면 할당 초과/미달 가능성이 있는 리소스를 보고하고 클러스터의  용량 부족에 대해 경고한다. Popeye를 사용하면 죽은 리소스와 사용되지 않는 리소스, 포트 불일치, RBAC 규칙, 메트릭 사용률 등을 쉽게 식별할 수

 

Popeyes는 읽기 전용 도구이며 클러스터를 보호하고 정리하는 데 도움이 되는 정보만 검색하며 Kubernetes  클러스터의 리소스를 수정하거나 삭제하지 않는다.

 

자세한 정보는 https://github.com/derailed/popeye 를 참고한다.

 

설치

 

# wget https://github.com/derailed/popeye/releases/download/v0.10.0/popeye_Linux_x86_64.tar.gz
# tar xvfz popeye_Linux_x86_64.tar.gz
# chmox +x popeye
# mv popeye /usr/local/bin/.

 

설치가 정상적인지 확인해보자.

# popeye version
 ___     ___ _____   _____                       K          .-'-.
| _ \___| _ \ __\ \ / / __|                       8     __|      `\
|  _/ _ \  _/ _| \ V /| _|                         s   `-,-`--._   `\
|_| \___/_| |___| |_| |___|                       []  .->'  a     `|-'
  Biffs`em and Buffs`em!                            `=/ (__/_       /
                                                      \_,    `    _)
                                                         `----;  |
Version:   0.10.0
Commit:    d1d807b721b0393e7b69e970bdf947895c61d8f4
Date:      2022-04-14T15:51:42Z
Logs:      /tmp/popeye.log

 

실행

 

기본 Popeye만 입력하면 모든 노드와 네임스페이스에서 테스트를 실행한다.

# popeye

 ___     ___ _____   _____                                                      K          .-'-.
| _ \___| _ \ __\ \ / / __|                                                      8     __|      `\
|  _/ _ \  _/ _| \ V /| _|                                                        s   `-,-`--._   `\
|_| \___/_| |___| |_| |___|                                                      []  .->'  a     `|-'
  Biffs`em and Buffs`em!                                                          `=/ (__/_       /
                                                                                    \_,    `    _)
                                                                                       `----;  |


GENERAL [KIND-KIND]
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Connectivity...................................................................................✅
  · MetricServer...................................................................................✅


CLUSTER (1 SCANNED)                                                          💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Version........................................................................................✅
    ✅ [POP-406] K8s version OK.


CLUSTERROLES (80 SCANNED)                                                  💥 0 😱 0 🔊 16 ✅ 64 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · admin..........................................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · argocd-application-controller..................................................................✅
  · argocd-server..................................................................................✅
  · cluster-admin..................................................................................✅
  · edit...........................................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · goldilocks-controller..........................................................................✅
  · goldilocks-dashboard...........................................................................✅
  · ingress-nginx..................................................................................✅
  · ingress-nginx-admission........................................................................✅
  · kindnet........................................................................................✅
  · kubeadm:get-nodes..............................................................................✅
  · local-path-provisioner-role....................................................................✅
...
STATEFULSETS (3 SCANNED)                                                       💥 0 😱 3 🔊 0 ✅ 0 0٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · argocd/argocd-application-controller...........................................................😱
    🐳 argocd-application-controller
      😱 [POP-106] No resources requests/limits defined.
      🔊 [POP-108] Unnamed port 8082.
  · keycloak/keycloak..............................................................................😱
    🐳 keycloak
      😱 [POP-106] No resources requests/limits defined.
  · keycloak/keycloak-postgresql...................................................................😱
    🐳 postgresql
      😱 [POP-107] No resource limits defined.


SUMMARY
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
Your cluster score: 81 -- B
                                                                                o          .-'-.
                                                                                 o     __| B    `\
                                                                                  o   `-,-`--._   `\
                                                                                 []  .->'  a     `|-'
                                                                                  `=/ (__/_       /
                                                                                    \_,    `    _)
                                                                                       `----;  |

 

특정 Namespace만 하려면 아래와 같이 실행한다.

# popeye -n test

 ___     ___ _____   _____                                                      K          .-'-.
| _ \___| _ \ __\ \ / / __|                                                      8     __|      `\
|  _/ _ \  _/ _| \ V /| _|                                                        s   `-,-`--._   `\
|_| \___/_| |___| |_| |___|                                                      []  .->'  a     `|-'
  Biffs`em and Buffs`em!                                                          `=/ (__/_       /
                                                                                    \_,    `    _)
                                                                                       `----;  |


GENERAL [KIND-KIND]
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Connectivity...................................................................................✅
  · MetricServer...................................................................................✅

...
PODS (2 SCANNED)                                                               💥 0 😱 2 🔊 0 ✅ 0 0٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · test/hamster-78f9dcdd4c-gwxg4..................................................................😱
      😱 [POP-109] CPU Current/Request (504m/100m) reached user 80% threshold (504%).
    🔊 [POP-206] No PodDisruptionBudget defined.
    😱 [POP-300] Using "default" ServiceAccount.
    😱 [POP-301] Connects to API Server? ServiceAccount token is mounted.
    🐳 hamster
      😱 [POP-107] No resource limits defined.
      😱 [POP-102] No probes defined.
  · test/hamster-78f9dcdd4c-xn5jb..................................................................😱
      😱 [POP-109] CPU Current/Request (501m/100m) reached user 80% threshold (501%).
    🔊 [POP-206] No PodDisruptionBudget defined.
    😱 [POP-300] Using "default" ServiceAccount.
    😱 [POP-301] Connects to API Server? ServiceAccount token is mounted.
    🐳 hamster
      😱 [POP-107] No resource limits defined.
      😱 [POP-102] No probes defined.


CLUSTER (1 SCANNED)                                                          💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Version........................................................................................✅
    ✅ [POP-406] K8s version OK.

 

Popeye에서 나오는 결과는 반드시 적용해야 하는것은 아니지만 따라야 할 모범 사례를 제시한다.

• 태그가 지정된 Docker  이미지 사용
• 리소스 요청 및  제한 설정
• 불필요한 ServiceAccount mount
• 설정된 Resource Request보다 사용량이 많은 workload

또한 다음과 같은 장애들을 일목요연하게 보여준다.

• Crash 된 Pod 발견
• Endpoint가 없는 Service 발견
• desired 개수와 다른 Deployment나 ReplicaSet 발견
• Pending중인 PVC

파일로 저장하려면 아래와 같이 한다.

# popeye -n test --save
/tmp/popeye/sanitizer_kind-kind_1652622413207737700.txt

 

읽기전용이기 때문에 개발기나 운영기에서 주기적으로 체크하면 좋을 거 같다.