Clusterpedia로 멀티 클러스터 조회하기

Clusterpedia는 클러스터의 위키피다이며 Kubernetes 리소스를 자체 데이터베이스에 동기화하여 여러 클러스터에 대한 강력하고 복잡한 검색을 허용하는 CNCF 프로젝트이다. 또한 Clusterpedia는 다른 리소스 버전과 호환된다.

 

아키텍처

https://github.com/clusterpedia-io/clusterpedia

 

• Clusterpedia APIServer : Kubernetes APIServer를 등록 및 통합출입을 통한 서비스 제공
• ClusterSynchro Manager : 클러스터 동기화를 관리
• Storage Layer : Cluster 의 상태를 저장하기 위한 Storage layer interface
• 스토리지 구성요소 : MySQL , PostgreSQL , Redis 또는 기타 그래프 데이터베이스 와 같은 특정 스토리지

 

설치

git clone https://github.com/clusterpedia-io/clusterpedia.git
cd clusterpedia
git checkout v0.5.0

1) Storage Component 설치

cd ./deploy/internalstorage/postgres

postgres 스토리지는 로컬 PV를 사용하여 배포된다. 아래와 같이 설치될 노드를 선택한다. ( storageclass를 사용하는 경우에는 아래 절차가 아닌 수동으로  PVC 맵핑을 한다. )

export STORAGE_NODE_NAME=node1
sed "s|__NODE_NAME__|$STORAGE_NODE_NAME|g" `grep __NODE_NAME__ -rl ./templates` > clusterpedia_internalstorage_pv.yaml

# kubectl apply -f .
namespace/clusterpedia-system created
configmap/clusterpedia-internalstorage created
service/clusterpedia-internalstorage-postgres created
persistentvolumeclaim/internalstorage-postgres created
deployment.apps/clusterpedia-internalstorage-postgres created
persistentvolume/clusterpedia-internalstorage-postgres created
job.batch/check-node1-postgres-local-pv-dir created
secret/internalstorage-password created

2) Clusterpedia 설치

cd ../../../
kubectl apply -f ./deploy

3) 설치 확인

# kubectl -n clusterpedia-system get pods
NAME                                                     READY   STATUS    RESTARTS   AGE
clusterpedia-apiserver-7f7fcbf8f6-hgh54                  1/1     Running   0          40s
clusterpedia-clustersynchro-manager-5c5dcbbd68-m5f5f     1/1     Running   0          40s
clusterpedia-controller-manager-56dfbf5f9d-l8nmc         1/1     Running   0          40s
clusterpedia-internalstorage-postgres-6cc9d67786-k2dbt   1/1     Running   0          3m

 

Clusterpedia 제거

kubectl delete -f ./deploy/clusterpedia_apiserver_apiservice.yaml
kubectl delete -f ./deploy/clusterpedia_apiserver_deployment.yaml
kubectl delete -f ./deploy/clusterpedia_clustersynchro_manager_deployment.yaml
kubectl delete -f ./deploy/clusterpedia_apiserver_rbac.yaml
kubectl delete -f ./deploy/cluster.clusterpedia.io_pediaclusers.yaml

 

설정

Clusterpedia는 ediaCluster리소스 사용하여 동기화하고 검색해야 하는 클러스터를 나타낸다.

apiVersion: cluster.clusterpedia.io/v1alpha2
kind: PediaCluster
metadata:
  name: demo1
spec:
  apiserver: https://10.6.101.100:6443
  kubeconfig:
  caData:
  tokenData:
  certData:
  keyData:
  syncResources: []
  syncAllCustomResources: false
  syncResourcesRefName: ""

설정하기 위한 가장 쉬운 방법은 K8S 설치 시 설정되는 ./kube/config 파일을 등록하는 것이다. 등록 방법은 아래와 같다.

# base64 -w 0 ~/.kube/config
YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIGNlcnRpZmljYXRlLWF1dGhvcml0eS1kYXRhOiBMUzB0TFMxQ1JVZEpUaUJEUlZKVVNVWkpRMEZVUlMwdExTMHRDazFKU1VNdmFrTkRRV1ZoWjBGM1NVSkJaMGxDUVVSQlRrSm5hM0ZvYTJsSE9YY3dRa0ZSYzBaQlJFRldUVkpOZDBWUldVUldVVkZFUlhkd2NtUlhTbXdLWTIwMWJHUkhWbnBOUWpSWVJGUkplRTFFYTNsT1JFVjNUVlJOZVU1R2IxaEVWRTE0VFVScmVVMXFSWGROVkUxNVRrWnZkMFpVUlZSTlFrVkhRVEZWUlFwQmVFMUxZVE5XYVZwWVNuVmFXRkpzWTNwRFEwRlRTWGRFVVZsS1MyOWFTV2gyWTA1QlVVVkNRbEZCUkdkblJWQkJSRU5EUVZGdlEyZG5SVUpCVHk5VENuWnRhMVU1YmsxdVVsUklUM2x2SzNoamRGUkpZMGxQWW5NemMwRjVjVEkyZGpSUVlrVnRiM1pXTTJ4UE9WUXdNVEYyY0U5NlMwcHlPVUZ4ZVZaTVJuWUtWWEZCUkhCVGFrTTNXWGQzTW5ad1NsZDNiREV5U2xCdlVtMXhaMUZCU0ZOa1lsSnBVM0JEVERSdWRqbHZSMjVWT1dJMmRsbFdTeTlpUml0a1VWRkNTQXBuUTFoNk5uWm9UR1k0V21kMk4ydFVRMkpCZGtGUGFFOU9TbFUzTWxsWVRFOHpUMGxaUWpKdmExTkNSR0ZWVWpOdk5ucHdaR1ZXVGt0NVYwRXlOVkEzQ2tSb2JrOHlUazAxUXpscFJFUnFUVFJMWTJGVGEzSlBTa0p2YlVsc1NIRlpSalJ3VlhkVFRsRnZjR1ZHUlZSeVozWnpjVGt3U2tzMllVSlZTMHQ1YWpZS0syTkdkakkzUzBrNEsxWk1VRXRhU1RFMmMyNU1ibmcyUlhSVGF6WnRaakpYVEhkSlpsaHlRbGd3UkVzdllYQkVRMDE1UjJwRWIyZENhR3BKU1Zob1ZBcDJialZRWm5kRldVTnNkR1pGVEVoS1NrZFZRMEYzUlVGQllVNWFUVVpqZDBSbldVUldVakJRUVZGSUwwSkJVVVJCWjB0clRVRTRSMEV4VldSRmQwVkNDaTkzVVVaTlFVMUNRV1k0ZDBoUldVUldVakJQUWtKWlJVWkpWRGhMUkhkQ2JVVnZNSGxhZFVGRVpraGtLelExTDNaRll6ZE5RbFZIUVRGVlpFVlJVVThLVFVGNVEwTnRkREZaYlZaNVltMVdNRnBZVFhkRVVWbEtTMjlhU1doMlkwNUJVVVZNUWxGQlJHZG5SVUpCVDBGNVZIUTRTM1pGTjBkdlJFaFFUMDlwZGdveVIySTJXV1ZzVVU1S2NVTXphMWRJT1hjMU5URk5hR1p2UzNaaU0yMVZhVVY2WlZNd09VTndaVVFyVEZoNVpubHFRemhaWWtKeFFqWlhTRmhOWldNckNucFBkRE5QYXpSWVYwRm1aVlZaVFhoT1ExRkpibGM0Y2pJNGNtWm5ibEVyYzFOQ2RIUXllRVJRTjFSWlkwOW9OVlpHWmtJMkszSnRUbUZUYmxaMU5qZ0tTRkZ4ZGxGTU5FRlhiVmhrUjA5alJXTkJSVGhZZGtkaU9XaHdTalZOY2tSSGR6UTBVVFl5T0c5WWF6WjBOMDFhV1RGT01VTlFkVzlIWjFWbVMxTjNiZ28xTVVGV1JURk9WVmROVjB0RVFYaGFhMkk0YkVodlIzVldhREZ6V21kM1NuSlJRalI1Y2xoMWNteEdOMFkyYlZSbFltNHJjRFZLTTB0b1QwVjRLemxzQ2pGWGRrd3diV2t4TDFKMmJWSktObTExWW10aldVd3pOMUZKV2pJMVlYZHlhRVpNTjBaMWVqTlJTVEZxVFRkWU1IWkVUMlZVTTJWdVZVRkNaVzVTTVM4S1VubG5QUW90TFMwdExVVk9SQ0JEUlZKVVNVWkpRMEZVUlMwdExTMHRDZz09CiAgICBzZXJ2ZXI6IGh0dHBzOi8vMTAuNi4xMDAuMTA6NjQ0MwogIG5hbWU6IGt1YmVybmV0ZXMKY29udGV4dHM6Ci0gY29udGV4dDoKICAgIGNsdXN0ZXI6IGt1YmVybmV0ZXMKICAgIHVzZXI6IGt1YmVybmV0ZXMtYWRtaW4KICBuYW1lOiBrdWJlcm5ldGVzLWFkbWluQGt1YmVybmV0ZXMKY3VycmVudC1jb250ZXh0OiBrdWJlcm5ldGVzLWFkbWluQGt1YmVybmV0ZXMKa2luZDogQ29uZmlnCnByZWZlcmVuY2VzOiB7fQp1c2VyczoKLSBuYW1lOiBrdWJlcm5ldGVzLWFkbWluCiAgdXNlcjoKICAgIGNsaWVudC1jZXJ0aWZpY2F0ZS1kYXRhOiBMUzB0TFMxQ1JVZEpUaUJEUlZKVVNVWkpRMEZVUlMwdExTMHRDazFKU1VSSlZFTkRRV2R0WjBGM1NVSkJaMGxKV2s0eVNscE5TbnAwU21kM1JGRlpTa3R2V2tsb2RtTk9RVkZGVEVKUlFYZEdWRVZVVFVKRlIwRXhWVVVLUVhoTlMyRXpWbWxhV0VwMVdsaFNiR042UVdWR2R6QjVUVlJCTlUxcVVYaE5SRVY2VFdwU1lVWjNNSGxOYWtFMVRXcFJlRTFFUlhwTmFtaGhUVVJSZUFwR2VrRldRbWRPVmtKQmIxUkViazQxWXpOU2JHSlVjSFJaV0U0d1dsaEtlazFTYTNkR2QxbEVWbEZSUkVWNFFuSmtWMHBzWTIwMWJHUkhWbnBNVjBackNtSlhiSFZOU1VsQ1NXcEJUa0puYTNGb2EybEhPWGN3UWtGUlJVWkJRVTlEUVZFNFFVMUpTVUpEWjB0RFFWRkZRVFZEUkdSYVdIcEliMXAxVVRKeFJEZ0tTakpRZGtWdWEyTk1UV05RVG14RE1DOVRTR1YzV25kME5FRjRLM2RDWTFSSVJ6aGpWakJhZUZSYVQwdDNPSFJ4UWxrMk1tcGtOM1p4VkdoeFRWbHdad3AyYzNwSFVXeHlXbGRyZHpSUmFrUldORnBLY1dSbFRITkRVV3BqZUZsa05Ea3JSalEyYkVsS1VUSjVjRXhTUjBkb2NGTlpZMlYzWkdOTVkweHNTamRIQ21wRl...

위의 결과를 이용하여 PediaCluster CRD 를 생성한다.

apiVersion: cluster.clusterpedia.io/v1alpha2
kind: PediaCluster
metadata:
  name: cluster1
spec:
  apiserver: "https://10.30.43.43:6443"
  kubeconfig: YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1...
  syncResources: []

이제 만든 CRD를 생성하여 정보가 잘 보이는지 확인한다.

# k create -f p.yaml
# kubectl get pediacluster
NAME       READY   VERSION   APISERVER
cluster1   True    v1.25.0   https://10.30.43.43:6443

 

Clusterpedia 액세스

URL을 사용하여 Clusterpedia 리소스에 액세스할 수 있지만 kubectl을 사용하여 더 편리하게 쿼리하려면 kubeconfig 클러스터를 구성해야 한다.

curl -sfL https://raw.githubusercontent.com/clusterpedia-io/clusterpedia/v0.5.0/hack/gen-clusterconfigs.sh | sh -

이제 kubectl 로 멀티 클러스터 조회가 가능하다.

kubectl --cluster clusterpedia api-resources

 

Cluster Resource 동기화

Clusterpedia의 주요 기능은 여러 클러스터에서 리소스에 대한 복잡한 검색을 제공하는 것이다. Clusterpedia는 PediaCluster리소스를 사용하여 클러스터에서 복잡한 검색을 지원해야 하는 리소스를 지정하고 이러한 리소스를 실시간으로 Storage Component Layuer에 동기화 한다.

apiVersion: cluster.clusterpedia.io/v1alpha2
kind: PediaCluster
metadata:
  name: cluster1
spec:
  apiserver: "https://10.30.43.43:6443"
  syncResources:
  - group: apps
    resources:
     - deployments
  - group: ""
    resources:
     - pods
     - configmaps

syncResources를 추가한 후 적용해 보자. ( cluster2도 동일하게 적용한다. )

 

# kubectl get --raw="/apis/clusterpedia.io/v1beta1/resources/apis/apps" | jq
{
  "kind": "APIGroup",
  "apiVersion": "v1",
  "name": "apps",
  "versions": [
    {
      "groupVersion": "apps/v1",
      "version": "v1"
    },
    {
      "groupVersion": "apps/v1beta2",
      "version": "v1beta2"
    },
    {
      "groupVersion": "apps/v1beta1",
      "version": "v1beta1"
    }
  ],
  "preferredVersion": {
    "groupVersion": "apps/v1",
    "version": "v1"
  }
}

Clusterpedia는 Deployment 의 v1, v1beta1, v1beta2  세 가지 버전을 지원한다.

 

다중클러스터 검색

kubectl에서 현재 검색할 수 있는 리소스를 확인해 보자.

# kubectl --cluster clusterpedia api-resources
NAME          SHORTNAMES   APIVERSION   NAMESPACED   KIND
configmaps    cm           v1           true         ConfigMap
pods          po           v1           true         Pod
deployments   deploy       apps/v1      true         Deployment

이제 deployments 를 검색해 보자.

# kubectl --cluster clusterpedia get deployments -A
NAMESPACE             CLUSTER    NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
clusterpedia-system   cluster1   clusterpedia-internalstorage-postgres   1/1     1            1           95m
kube-system           cluster1   coredns                                 2/2     2            2           71d
kube-system           cluster1   calico-kube-controllers                 1/1     1            1           71d
kube-system           cluster2   metrics-server                          1/1     1            1           23d
...
kube-system           cluster2   calico-kube-controllers                 1/1     1            1           44d
kube-system           cluster2   coredns                                 2/2     2            2           44d
...

특정 cluster만 검색하고자 하는 경우는

# kubectl --cluster clusterpedia get deployments -A -l "search.clusterpedia.io/clusters in (cluster1)"
kube-system           cluster1   coredns                                 2/2     2            2           71d
kube-system           cluster1   calico-kube-controllers                 1/1     1            1           71d
kube-system           cluster2   metrics-server                          1/1     1            1           23d

 

단일 클러스터만 검색하는 경우는 다음과 같이 사용할 수도 있다.

# kubectl --cluster cluster1 get deployments -n kube-system
CLUSTER    NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
cluster1   calico-kube-controllers   1/1     1            1           71d
cluster1   coredns                   2/2     2            2           71d
cluster1   metrics-server            1/1     1            1           34d