k8s孤儿Pod

643次阅读
没有评论

共计 3121 个字符,预计需要花费 8 分钟才能阅读完成。

k8s孤儿Pod

今天的云挺饱满的,可惜下起了太阳雨……..

k8s孤儿Pod

最近博主在清理pod时,看到kubelet日志一直在刷一种日志

2023 08 03101:41:27.670582591Z W0803 01:41:27 6/0427 3388387 kubelet getters.go:297] Path "/var/lib/kubelet/pods/937d53d9-e49d-4cd4-975c-56b252a826ae/volumes" does not exist 
2023 08 03T01:41:27.670588854Z W0803 01:41:27.670448 3388387 kubelet getters.go:297] Path "/var/lib/kubelet/pods/ab5c894a-aa6c-490e-970a-15f419ae4dd9/volumes" does not exist 
2023 08 03T01:41:27.6705930947 W0803 01:41:27.670456 3388387 kubelet getters.go:297] Path "/var/ib/kubelet/pods/ab5c894a-aa6c-490e -970a-15f419ae4dd9/volumes" does not exist 
2023 08 03T01:41:27.670597180Z W0803 01:41:27.670477 3388387 kubelet getters.go:297] Path "/var/lib/kubelet/pods/dbf4672d-00cf-4bc7-9bdf-6fe8e5e82fb4/volumes" does not exist 
2023 08 03T01:41:27.670601213Z W0803 01:41:27.670485 3388387 kubelet getters.go:297] Path "/ar/ib/kubelet/pods/dbf4672d-00cf-4bc7-9bdf-6fe8e5e82fb4/volumes" does not exist There were a total of 3 errors similar to this. Turn up verbosity to see them.
2023 08 03T01:41:27.6706053287 W0803 01:41:27.670497 3388387 kubelet volumes.go:154] orphaned pod "937d53d9-e49d-4cd4-975c-56b252a826ae" found, but volume subpaths are still present on disk:

通过日志中的Pod uid去寻找pod,已无法找到,说明pod已删除,但是目前看还有资源未释放掉,尝试手动删除

[root@XXXXX pods]# rm -rf /var/lib/kubelet/pods/035b275f-ffb2-43debaa6-64ee73e55b35
rm: cannot remove '/var/lib/kubelet/pods/035b275f-ffb2-43debaa6-64ee73e55b35/volume -subpaths/multus-cfg/kube-multus/2': Device or resource busy

grep以下mount信息看看

[root@XXXXX pods]# mount -l | grep '035b275f-ffb2-43debaa6-64ee73e55b35'
/dev/mapper/rootvg-var on /var/lib/kubelet/pods/035b275f-ffb2-43debaa6-64ee73e55b35/volume subpaths/multus-cfg/kube-multus/2 type xis (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)

看来得先umount后再删掉

[root@XXXXX pods]# umount /var/lib/kubelet/pods/035b275f-ffb2-43debaa6-64ee73e55b35
[root@XXXXX pods]# mount -l | grep 035b275f-ffb2-43debaa6-64ee73e55b35
[root@XXXXX pods]# rm -rf umount /var/lib/kubelet/pods/035b275f-ffb2-43debaa6-64ee73e55b35

此时再看kubelet日志已少一条该类型报错,那剩余的类似错误依照这个方式处理即可

google了一番,说是kubelet再回收orphaned pod时,遇到pod内还挂载volumePath或subPath就会报这个错

// cleanupOrphanedPodDirs removes the volumes of pods that should not be
// running and that have no containers running.  Note that we roll up logs here since it runs in the main loop.
func (kl *Kubelet) cleanupOrphanedPodDirs(pods []*v1.Pod, runningPods []*kubecontainer.Pod) error {
        // If there are still volume directories, do not delete directory
        volumePaths, err := kl.getPodVolumePathListFromDisk(uid)
        if err != nil {
            orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("orphaned pod %q found, but error %v occurred during reading volume dir from disk", uid, err))
            continue
        }
        if len(volumePaths) > 0 {
            orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("orphaned pod %q found, but volume paths are still present on disk", uid))
            continue
        }

        // If there are any volume-subpaths, do not cleanup directories
        volumeSubpathExists, err := kl.podVolumeSubpathsDirExists(uid)
        if err != nil {
            orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("orphaned pod %q found, but error %v occurred during reading of volume-subpaths dir from disk", uid, err))
            continue
        }
        if volumeSubpathExists {
            orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("orphaned pod %q found, but volume subpaths are still present on disk", uid))
            continue
        }
}

如果是这样,那为什么pod清理时,为什么会无法释放挂载点?这下不得不了解下pod的释放流程了。。。

正文完
 
xadocker
版权声明:本站原创文章,由 xadocker 2023-08-04发表,共计3121字。
转载说明:除特殊说明外本站文章皆由CC-4.0协议发布,转载请注明出处。
评论(没有评论)