Pod 생성 시 FailedMount : no space left on device 오류

Pod의 상태를 보니 아래와 같이 ContainerCreating 상태였다.

 

Pod상태

# k get po -n test
NAME                         READY   STATUS              RESTARTS   AGE
sampleapp-6c65c5cdb7-djktc   1/1     Running             0          33d
sampleapp-7c56d989f9-zvh5c   0/1     ContainerCreating   0          2m51s

 

에러 메시지

아래와 같이 FailedMount : no space left on device 오류 이벤트가 발생한다.

## k describe pod -n test sampleapp-7c56d989f9-zvh5c
Name:           sampleapp-7c56d989f9-zvh5c
Namespace:      test
...
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/sampleapp-7c56d989f9
Containers:
  sampleapp:
    Container ID:
    Image:          nginx
    Image ID:
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  128
    Requests:
      cpu:        1
      memory:     128
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vh4zz (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-vh4zz:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                   From               Message
  ----     ------       ----                  ----               -------
  Warning  FailedMount  6m4s                  kubelet            MountVolume.SetUp failed for volume "kube-api-access-vh4zz" : write /var/lib/kubelet/pods/1f6fc99f-dd5f-4994-9fc8-33ec6a7ba5eb/volumes/kubernetes.io~projected/kube-api-access-vh4zz/..2022_08_17_07_51_41.2438688674/ca.crt: no space left on device
  Warning  FailedMount  6m3s                  kubelet            MountVolume.SetUp failed for volume "kube-api-access-vh4zz" : write /var/lib/kubelet/pods/1f6fc99f-dd5f-4994-9fc8-33ec6a7ba5eb/volumes/kubernetes.io~projected/kube-api-access-vh4zz/..2022_08_17_07_51_42.1805949476/namespace: no space left on device
  Normal   Scheduled    6m2s                  default-scheduler  Successfully assigned test/sampleapp-7c56d989f9-zvh5c to node2
  Warning  FailedMount  6m2s                  kubelet            MountVolume.SetUp failed for volume "kube-api-access-vh4zz" : write /var/lib/kubelet/pods/1f6fc99f-dd5f-4994-9fc8-33ec6a7ba5eb/volumes/kubernetes.io~projected/kube-api-access-vh4zz/..2022_08_17_07_51_43.3767916300/ca.crt: no space left on device
  Warning  FailedMount  6m                    kubelet            MountVolume.SetUp failed for volume "kube-api-access-vh4zz" : write /var/lib/kubelet/pods/1f6fc99f-dd5f-4994-9fc8-33ec6a7ba5eb/volumes/kubernetes.io~projected/kube-api-access-vh4zz/..2022_08_17_07_51_45.3037654241/ca.crt: no space left on device
  Warning  FailedMount  5m56s                 kubelet            MountVolume.SetUp failed for volume "kube-api-access-vh4zz" : write /var/lib/kubelet/pods/1f6fc99f-dd5f-4994-9fc8-33ec6a7ba5eb/volumes/kubernetes.io~projected/kube-api-access-vh4zz/..2022_08_17_07_51_49.1833876226/token: no space left on device
  Warning  FailedMount  5m48s                 kubelet            MountVolume.SetUp failed for volume "kube-api-access-vh4zz" : write /var/lib/kubelet/pods/1f6fc99f-dd5f-4994-9fc8-33ec6a7ba5eb/volumes/kubernetes.io~projected/kube-api-access-vh4zz/..2022_08_17_07_51_57.3311206179/ca.crt: no space left on device
  Warning  FailedMount  5m32s                 kubelet            MountVolume.SetUp failed for volume "kube-api-access-vh4zz" : write /var/lib/kubelet/pods/1f6fc99f-dd5f-4994-9fc8-33ec6a7ba5eb/volumes/kubernetes.io~projected/kube-api-access-vh4zz/..2022_08_17_07_52_13.10725485/token: no space left on device
  Warning  FailedMount  5m                    kubelet            MountVolume.SetUp failed for volume "kube-api-access-vh4zz" : write /var/lib/kubelet/pods/1f6fc99f-dd5f-4994-9fc8-33ec6a7ba5eb/volumes/kubernetes.io~projected/kube-api-access-vh4zz/..2022_08_17_07_52_45.2191982976/ca.crt: no space left on device
  Warning  FailedMount  114s (x2 over 3m56s)  kubelet            (combined from similar events): MountVolume.SetUp failed for volume "kube-api-access-vh4zz" : write /var/lib/kubelet/pods/1f6fc99f-dd5f-4994-9fc8-33ec6a7ba5eb/volumes/kubernetes.io~projected/kube-api-access-vh4zz/..2022_08_17_07_55_51.2323841164/ca.crt: no space left on device
  Warning  FailedMount  106s (x2 over 4m1s)   kubelet            Unable to attach or mount volumes: unmounted volumes=[kube-api-access-vh4zz], unattached volumes=[kube-api-access-vh4zz]: timed out waiting for the condition

 

원인

Pod의 Resource 사용량을 제한하기 위해 Quota를 아래와 같이 설정하였다.

...
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: sampleapp
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: sampleapp
        ports:
        - containerPort: 80
          protocol: TCP
        resources:
          limits:
            cpu: "1"
            memory: "128"
...

128Mi의 메모리를 설정을 하고자 했는데 사용자의 실수로 128로 지정하여 새롭게 생성될 Pod의 메모리 부족으로

ca 인증서 설치를 진행못하는 상태로 생성에 실패하였다.