Kubernetes 에서 Vault 활용

Vault

 

Valut는 HashiCorp 사에서 만든 오픈소스로 비밀 및 기타 민감한 데이터를 보호하기 위해 토큰, 암호, 인증서, 암호화 키에 대한 액세스를 보호, 저장 및 제어할 수 있는 기능을 가지고 있다.

 

이러한 Valut를 Kubernetes와 통합 하기 위해 크게 2가지 방법을 지원한다.

 

1. Valut Agent Injector

2. Valut Container Storage Interface (CSI)

 

두 가지 방식의 비교

 

출처 : https://www.hashicorp.com/blog/kubernetes-vault-integration-via-sidecar-agent-injector-vs-csi-provider

 

1. Valut Agent Injector

 

Vault 설치

 

# helm repo add hashicorp https://helm.releases.hashicorp.com

# helm repo update

# kubectl create namespace vault

# helm install vault hashicorp/vault --set ui.enabled=true -n vault

 

helm 으로 설치가 완료되었으면 설치된 내역을 조회해 보자.

 

# kubectl get all -n vault
NAME                                        READY   STATUS    RESTARTS   AGE
pod/vault-0                                 1/1     Running   0          85s
pod/vault-agent-injector-6cd49f8bbd-kr5k6   1/1     Running   0          85s

NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/vault                      ClusterIP   10.111.92.236   <none>        8200/TCP,8201/TCP   86s
service/vault-agent-injector-svc   ClusterIP   10.103.60.55    <none>        443/TCP             86s
service/vault-internal             ClusterIP   None            <none>        8200/TCP,8201/TCP   86s

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vault-agent-injector   1/1     1            1           86s

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/vault-agent-injector-6cd49f8bbd   1         1         1       86s

NAME                     READY   AGE
statefulset.apps/vault   1/1     86s

 

helm install --name=vault --set='server.dev.enabled=true' hashicorp/vault

위와 같이 설치 시 dev 모드로 설치되고 Vault 인스턴스가 자동으로 초기화되고 봉인 해제되고 인메모리 스토리지를 사용합니다. 운영기에서는 권장하지 않는다.

 

vault 설치 시 dev 모드로 설치하지 않으면 vault를 초기화하고 봉인 해제되어야 한다.

초기화를 하지 않으면 아래의 오류가 발생한다. ( Readiness probe failed 에러 발생)

Events:
  Type     Reason     Age              From               Message
  ----     ------     ----             ----               -------
  Normal   Scheduled  29s              default-scheduler  Successfully assigned vault/vault-0 to node2
  Normal   Pulling    28s              kubelet            Pulling image "hashicorp/vault:1.9.2"
  Normal   Pulled     17s              kubelet            Successfully pulled image "hashicorp/vault:1.9.2" in 11.034382036s
  Normal   Created    16s              kubelet            Created container vault
  Normal   Started    16s              kubelet            Started container vault
  Warning  Unhealthy  4s (x2 over 9s)  kubelet            Readiness probe failed: Key                Value​

아래와 같이 초기화 한다.

# kubectl exec -it vault-0 -n vault -- vault operator init
Unseal Key 1: l3g8myV0M07MBmk1vX095vVPYotmOTyLzDMDNHl1EbiC
Unseal Key 2: oUchcpBj2+4zUS+15AAEoPpYyzTVsg/KHOANtVnVu0ov
Unseal Key 3: WOOMaPOvxYJxsAK7pJDV2U6a72VRDbdNiydr7v1e88i5
Unseal Key 4: 8p16U3nbGinmZHH7BmX6sSRh0Lzq2nHb7wRIx20XOSE6
Unseal Key 5: rzXP15+IrCnDK26zohbH3l46OciLPM94Q+uoG0eTDIBM

Initial Root Token: s.FeqJzJn11wmq9HiUhHwskQzQ

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 keys to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

 

Valut에서 Secret 설정

 

Vault에서 secret를 설정하는 방식은 아래와 같다.

 

# kubectl exec -it vault-0 -n vault -- /bin/sh

# internal 경로에서 kv-v2 secret을 활성화한다.
/ $ vault secrets enable -path=internal kv-v2
Error enabling: Error making API request.

URL: POST http://127.0.0.1:8200/v1/sys/mounts/internal
Code: 503. Errors:

* Vault is sealed

# vault를 unsealed 한다. 키 3개 이상으로 해야 봉인이 해제된다.
$ vault operator unseal <unsealed key>


# 다시 kv-v2 secret을 활성화환다.
$ vault secrets enable -path=internal kv-v2
Error enabling: Error making API request.

URL: POST http://127.0.0.1:8200/v1/sys/mounts/internal
Code: 400. Errors:

* missing client token

# token 정보를 env에 설정한다.
# export VAULT_TOKEN=s.FeqJzJn11wmq9HiUhHwskQzQ
# vault secrets enable -path=internal kv-v2
Success! Enabled the kv-v2 secrets engine at: internal/

# internal/database/config 경로에 secret를 만든다.
$ vault kv put internal/database/config username="id" password="mypassword"

# secret 정의가 잘 되었는지 확인한다.
$ vault kv get internal/database/config
Key                Value
---                -----
created_time       2022-03-14T13:24:00.224931832Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1
/ $ vault kv get internal/database/config
======= Metadata =======
Key                Value
---                -----
created_time       2022-03-14T13:24:00.224931832Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    mypassword
username    id

 

Vault UI에서도 아래와 같이 확인됨을 볼 수 있다.

 

 

Kubernetes 인증 구성

 

Vault는 클라이언트가 Kubernetes 서비스 계정 토큰으로 인증할 수 있도록 하는 Kubernetes 인증 방법을 제공한다.

 

$ kubectl exec -it vault-0 -n vault -- /bin/sh
/ $ vault auth enable kubernetes

$ vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    disable_iss_validation=true \
    issuer="https://kubernetes.default.svc.cluster.local"
Success! Data written to: auth/kubernetes/config

$ vault policy write internal-app - <<EOF
path "internal/data/database/config" {
  capabilities = ["read"]
}
EOF
Success! Uploaded policy: internal-app


$ vault write auth/kubernetes/role/internal-app \
    bound_service_account_names=internal-app \
    bound_service_account_namespaces=vault \
    policies=internal-app \
    ttl=24h
Success! Data written to: auth/kubernetes/role/internal-app

 

이제 Kubernetes 서비스 계정 정의를 해보자.

 

# kubectl create sa internal-app -n vault

# kubectl get serviceaccounts -n valult
default                1         109m
internal-app           1         5s
vault                  1         22m
vault-agent-injector   1         22m

 

애플리케이션 실행

 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: vault
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/agent-inject-status: 'update'
        vault.hashicorp.com/role: 'internal-app'
        vault.hashicorp.com/agent-inject-secret-database-config.txt: 'internal/data/database/config'
        vault.hashicorp.com/agent-inject-template-database-config.txt: |
          {{- with secret "internal/data/database/config" -}}
          postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
          {{- end -}}
      labels:
        app: nginx
    spec:
      serviceAccountName: internal-app
      containers:
        - name: nginx
          image: nginx

 

실행 시 아래와 같이 Pending이 걸리고, 로그에 "context deadline exceeded" 에러가 발생한다면 

vault pod/nginx-64f6bc66c4-lfd9t 0/2 Init:0/1 0 35s 172.32.24.35 node2 <none> <none>

#  kubectl logs -n vault nginx-5d5c65dc7f-6w9kv -c vault-agent-init

[INFO]  auth.handler: authenticating
[ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=4m36.53s​

vault write auth/kubernetes/role/internal-app \
    bound_service_account_names=internal-app \
    bound_service_account_namespaces=vault \   <----- namespace 확인
    policies=internal-app \
    ttl=24h
Success! Data written to: auth/kubernetes/role/internal-app​

 

이제 아래와 같이 nginx pod에 secret 파일을 조회할 수 있다.

 

# kubectl exec -n vault nginx-5594948fc9-qpjdv -c nginx -- cat /vault/secrets/database-config.txt
postgresql://id:mypassword@postgres:5432/wizard