k8s中一些常用运维命令

382次阅读
没有评论

共计 12434 个字符，预计需要花费 32 分钟才能阅读完成。

k8s中一些常用运维命令

此处做个备忘录，记录下博主常用的一些k8s命令

集群

查看集群版本

[root@k8s-master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9", GitCommit:"94f372e501c973a7fa9eb40ec9ebd2fe7ca69848", GitTreeState:"clean", BuildDate:"2020-09-16T13:56:40Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9", GitCommit:"94f372e501c973a7fa9eb40ec9ebd2fe7ca69848", GitTreeState:"clean", BuildDate:"2020-09-16T13:47:43Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

查看集群信息

[root@k8s-master ~]# kubectl cluster-info
Kubernetes master is running at https://apiserver.demo:6443
KubeDNS is running at https://apiserver.demo:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

查看kubeconfig配置

[root@k8s-master ~]# kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://apiserver.demo:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

查看当前集群api-resources

[root@k8s-master ~]# kubectl api-resources
NAME                                                                           SHORTNAMES               APIGROUP                       NAMESPACED   KIND
bindings                                                                                                                               true         Binding
componentstatuses                                                              cs                                                      false        ComponentStatus
configmaps                                                                     cm                                                      true         ConfigMap
endpoints                                                                      ep                                                      true         Endpoints
events                                                                         ev                                                      true         Event
limitranges                                                                    limits                                                  true         LimitRange
namespaces                                                                     ns                                                      false        Namespace
nodes                                                                          no                                                      false        Node
persistentvolumeclaims                                                         pvc                                                     true         PersistentVolumeClaim
persistentvolumes                                                              pv                                                      false        PersistentVolume
pods                                                                           po                                                      true         Pod
podtemplates                                                                                                                           true         PodTemplate
replicationcontrollers                                                         rc                                                      true         ReplicationController
resourcequotas                                                                 quota                                                   true         ResourceQuota
secrets                                                                                                                                true         Secret
serviceaccounts                                                                sa                                                      true         ServiceAccount
services                                                                       svc                                                     true         Service
mutatingwebhookconfigurations                                                                           admissionregistration.k8s.io   false        MutatingWebhookConfiguration
validatingwebhookconfigurations                                                                         admissionregistration.k8s.io   false        ValidatingWebhookConfiguration
customresourcedefinitions                                                      crd,crds                 apiextensions.k8s.io           false        CustomResourceDefinition
####### 略

组件

查看组件状态

[root@k8s-master ~]# kubectl get cs
NAME                 STATUS      MESSAGE                                                                                     ERROR
scheduler            Unhealthy   Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
controller-manager   Unhealthy   Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refused
etcd-0               Healthy     {"health":"true"}

博主这里使用kubedm部署，这些组件的配置在/etc/kubernetes/manifests内

[root@k8s-master ~]# cd /etc/kubernetes/manifests/
[root@k8s-master manifests]# ll
total 16
-rw------- 1 root root 1882 Oct  1 12:46 etcd.yaml
-rw------- 1 root root 2730 Oct  1 12:46 kube-apiserver.yaml
-rw------- 1 root root 2598 Oct  1 12:46 kube-controller-manager.yaml
-rw------- 1 root root 1153 Oct  1 12:46 kube-scheduler.yaml

kubelet则在/var/lib/kubelete内

[root@k8s-master ~]# cd /var/lib/kubelet/
[root@k8s-master kubelet]# ll
total 16
-rw-r--r--  1 root root  782 Oct  1 12:45 config.yaml
-rw-------  1 root root   62 Oct  1 12:46 cpu_manager_state
drwxr-xr-x  2 root root   80 Jan 19 21:30 device-plugins
-rw-r--r--  1 root root  135 Oct  1 12:45 kubeadm-flags.env
drwxr-xr-x  2 root root  124 Oct  1 12:46 pki
drwxr-x---  2 root root    6 Oct  1 12:46 plugins
drwxr-x---  2 root root    6 Oct  1 12:46 plugins_registry
drwxr-x---  2 root root   26 Jan 19 21:30 pod-resources
drwxr-x--- 35 root root 4096 Jan 18 22:59 pods

kube-proxy的配置在configmap中配置

[root@k8s-master kubelet]# kubectl get cm -n kube-system kube-proxy
NAME         DATA   AGE
kube-proxy   2      110d

coredns的配置在configmap中配置

[root@k8s-master kubelet]# kubectl get cm -n kube-system coredns
NAME      DATA   AGE
coredns   1      110d

calico-config网络组件配置在configmap中配置

[root@k8s-master kubelet]# kubectl get cm -n kube-system calico-config
NAME            DATA   AGE
calico-config   4      110d

node

查看节点状态

[root@k8s-master ~]# kubectl get nodes -o wide --show-labels
NAME          STATUS   ROLES            AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME   LABELS
k8s-master    Ready    compute,master   110d   v1.18.9   192.168.44.151   <none>        CentOS Linux 7 (Core)   3.10.0-1127.el7.x86_64   docker://19.3.8     beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master,kubernetes.io/os=linux,node-role.kubernetes.io/compute=dedicated-middleware,node-role.kubernetes.io/master=
k8s-node-01   Ready    <none>           32h    v1.18.9   192.168.44.152   <none>        CentOS Linux 7 (Core)   3.10.0-1127.el7.x86_64   docker://19.3.8     beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node-01,kubernetes.io/os=linux
k8s-node-02   Ready    <none>           32h    v1.18.9   192.168.44.153   <none>        CentOS Linux 7 (Core)   3.10.0-1127.el7.x86_64   docker://19.3.8     beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node-02,kubernetes.io/os=linux

新增node节点

# 获取增加节点命令，24小时时效性
[root@k8s-master ~]# kubeadm token create --print-join-command
W0119 23:51:01.740531   43997 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm join apiserver.demo:6443 --token 8criyg.0wf5n8wvjdzc2bp1     --discovery-token-ca-cert-hash sha256:94d16a5811ec81c8bf3c74d1afabfac2400d6b9ea653f9e5aad7a451a0de675c

# 获取token列表
[root@k8s-master ~]# kubeadm token list
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
8criyg.0wf5n8wvjdzc2bp1   23h         2021-01-20T23:51:01+08:00   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token
jjxora.ry0indl22meve43d   23h         2021-01-20T23:50:57+08:00   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token

# 根据上面获取的命令，将节点加入集群，在节点中运行
[root@k8s-node-01 ~]# kubeadm join apiserver.demo:6443 --token 8criyg.0wf5n8wvjdzc2bp1     --discovery-token-ca-cert-hash sha256:94d16a5811ec81c8bf3c74d1afabfac2400d6b9ea653f9e5aad7a451a0de675c

新增master节点

# 获取节点加入命令
[root@k8s-master ~]# kubeadm token create --print-join-command
W0120 00:06:54.280183   61911 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm join apiserver.demo:6443 --token jgfnf1.uitgucls32c779pi     --discovery-token-ca-cert-hash sha256:94d16a5811ec81c8bf3c74d1afabfac2400d6b9ea653f9e5aad7a451a0de675c

# 更新并获取 certificate key
[root@k8s-master ~]# kubeadm init phase upload-certs --upload-certs
I0120 00:07:49.896924   62979 version.go:252] remote version is much newer: v1.26.1; falling back to: stable-1.18
W0120 00:07:50.621028   62979 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
0ab9a27424ba575c975258db6908e1dd2f5078d084a249cc1669f9272975ac03

# 新master节点上拼接上面两部操作的命令
[root@k8s-master-02 ~]# kubeadm join apiserver.demo:6443 --token jgfnf1.uitgucls32c779pi     --discovery-token-ca-cert-hash sha256:94d16a5811ec81c8bf3c74d1afabfac2400d6b9ea653f9e5aad7a451a0de675c --control-plane --certificate-key 0ab9a27424ba575c975258db6908e1dd2f5078d084a249cc1669f9272975ac03

配置节点调度

# 查看当前节点状态
[root@k8s-master ~]# kubectl get nodes
NAME          STATUS   ROLES            AGE    VERSION
k8s-master    Ready    compute,master   110d   v1.18.9
k8s-node-01   Ready    <none>           32h    v1.18.9
k8s-node-02   Ready    <none>           32h    v1.18.9

# 设置节点1不可调度
[root@k8s-master ~]# kubectl cordon k8s-node-01
node/k8s-node-01 cordoned
[root@k8s-master ~]# kubectl get nodes
NAME          STATUS                     ROLES            AGE    VERSION
k8s-master    Ready                      compute,master   110d   v1.18.9
k8s-node-01   Ready,SchedulingDisabled   <none>           32h    v1.18.9
k8s-node-02   Ready                      <none>           32h    v1.18.9

# 设置可调度
[root@k8s-master ~]# kubectl uncordon k8s-node-01
node/k8s-node-01 uncordoned
[root@k8s-master ~]# kubectl get nodes
NAME          STATUS   ROLES            AGE    VERSION
k8s-master    Ready    compute,master   110d   v1.18.9
k8s-node-01   Ready    <none>           32h    v1.18.9
k8s-node-02   Ready    <none>           32h    v1.18.9

配置节点排水

# 一般下线节点时，先设置不可调度，然后驱逐pod
# 驱逐pod
[root@k8s-master ~]# kubectl cordon k8s-node-01
[root@k8s-master ~]# kubectl drain --ignore-daemonsets --delete-emptydir-data k8s-node-01

删除节点

[root@k8s-master ~]# kubectl delete node k8s-node-01

节点资源使用率

# 该功能依赖于metrics-server组件
[root@k8s-master ~]# kubectl top nodes
NAME          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
k8s-master    882m         11%    3722Mi          48%
k8s-node-01   226m         11%    1000Mi          27%
k8s-node-02   180m         9%     610Mi           16%

pod

查看pod状态

[root@k8s-master ~]# kubectl get pod -n kube-system -o wide --show-labels
NAME                                       READY   STATUS    RESTARTS   AGE    IP               NODE          NOMINATED NODE   READINESS GATES   LABELS
calico-kube-controllers-5b8b769fcd-8hlzn   1/1     Running   24         110d   10.100.235.229   k8s-master    <none>           <none>            k8s-app=calico-kube-controllers,pod-template-hash=5b8b769fcd
calico-node-fwcss                          1/1     Running   24         110d   192.168.44.151   k8s-master    <none>           <none>            controller-revision-hash=b9dd4bd9f,k8s-app=calico-node,pod-template-generation=1
calico-node-m84rz                          1/1     Running   1          32h    192.168.44.153   k8s-node-02   <none>           <none>            controller-revision-hash=b9dd4bd9f,k8s-app=calico-node,pod-template-generation=1
calico-node-tvs89                          1/1     Running   1          32h    192.168.44.152   k8s-node-01   <none>           <none>            controller-revision-hash=b9dd4bd9f,k8s-app=calico-node,pod-template-generation=1
coredns-65556b4c97-dhkz4                   1/1     Running   6          26d    10.100.235.193   k8s-master    <none>           <none>            k8s-app=kube-dns,pod-template-hash=65556b4c97
etcd-k8s-master                            1/1     Running   24         110d   192.168.44.151   k8s-master    <none>           <none>            component=etcd,tier=control-plane
kube-apiserver-k8s-master                  1/1     Running   24         110d   192.168.44.151   k8s-master    <none>           <none>            component=kube-apiserver,tier=control-plane
kube-controller-manager-k8s-master         1/1     Running   25         110d   192.168.44.151   k8s-master    <none>           <none>            component=kube-controller-manager,tier=control-plane
kube-proxy-h4smb                           1/1     Running   2          25h    192.168.44.152   k8s-node-01   <none>           <none>            controller-revision-hash=949786769,k8s-app=kube-proxy,pod-template-generation=1
kube-proxy-rf688                           1/1     Running   2          25h    192.168.44.151   k8s-master    <none>           <none>            controller-revision-hash=949786769,k8s-app=kube-proxy,pod-template-generation=1
kube-proxy-wtr4c                           1/1     Running   2          25h    192.168.44.153   k8s-node-02   <none>           <none>            controller-revision-hash=949786769,k8s-app=kube-proxy,pod-template-generation=1
kube-scheduler-k8s-master                  1/1     Running   24         110d   192.168.44.151   k8s-master    <none>           <none>            component=kube-scheduler,tier=control-plane
metrics-server-86499f7fd8-pdw6d            1/1     Running   4          10d    10.100.235.212   k8s-master    <none>           <none>            k8s-app=metrics-server,pod-template-hash=86499f7fd8
nfs-client-provisioner-df46b8d64-jwgd4     1/1     Running   24         110d   10.100.235.237   k8s-master    <none>           <none>            app=nfs-client-provisioner,pod-template-hash=df46b8d64

查看pod状态并排序

# 按启动时间降序
[root@k8s-master ~]# kubectl get pods --sort-by=.metadata.creationTimestamp

# 按启动时间升序，其实就是配合tac反转
[root@k8s-master ~]# kubectl get pods --sort-by=metadata.creationTimestamp --no-headers | tac

# 按pod重启次数升序排序
[root@k8s-master ~]# kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount'

查看pod资源利用率

[root@k8s-master ~]# kubectl top pod -n kube-system
NAME                                       CPU(cores)   MEMORY(bytes)
calico-kube-controllers-5b8b769fcd-8hlzn   3m           10Mi
calico-node-fwcss                          83m          58Mi
calico-node-m84rz                          50m          60Mi
calico-node-tvs89                          40m          60Mi
coredns-65556b4c97-dhkz4                   7m           15Mi
etcd-k8s-master                            46m          177Mi
kube-apiserver-k8s-master                  130m         619Mi
kube-controller-manager-k8s-master         55m          77Mi
kube-proxy-h4smb                           11m          19Mi
kube-proxy-rf688                           1m           16Mi
kube-proxy-wtr4c                           11m          19Mi
kube-scheduler-k8s-master                  7m           22Mi
metrics-server-86499f7fd8-pdw6d            5m           21Mi
nfs-client-provisioner-df46b8d64-jwgd4     2m           11Mi

查看pod日志

# 如果一个pod只有一个容器
[root@k8s-master ~]# kubectl logs -n kube-system etcd-k8s-master

# 如果一个pod有多个容器，查看指定容器日志
[root@k8s-master ~]# kubectl logs -n monitoring prometheus-operator-5cd4d464cc-g2m9w
error: a container name must be specified for pod prometheus-operator-5cd4d464cc-g2m9w, choose one of: [prometheus-operator kube-rbac-proxy]

[root@k8s-master ~]# kubectl logs -n monitoring prometheus-operator-5cd4d464cc-g2m9w prometheus-operator

获取所有pod的Qos

# 之前在做k8s升级时，14-》16时，如果request不等于limit的话，升级时是会有应用重启的预期
[root@k8s-master ~]# kubectl get pods --all-namespaces -o custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace,QOS-CLASS:.status.qosClass
NAME                                                         NAMESPACE          QOS-CLASS
busybox                                                      default            BestEffort
dnsutils                                                     default            BestEffort

nginx-a-55c8c877d5-29smq                                     default            BestEffort
nginx-a-55c8c877d5-5s92q                                     default            BestEffort

清理k8s异常pod

# 清理evicted状态pod
[root@k8s-master ~]# kubectl get pods --all-namespaces -o wide | grep Evicted | awk '{print $1,$2}' | xargs -L1 kubectl delete pod -n 

# 清理error状态pod
[root@k8s-master ~]# kubectl get pods --all-namespaces -o wide | grep Error | awk '{print $1,$2}' | xargs -L1 kubectl delete pod -n

# 清理compete状态pod
[root@k8s-master ~]# kubectl get pods --all-namespaces -o wide | grep Completed | awk '{print $1,$2}' | xargs -L1 kubectl delete pod -n

强制删除pod

[root@k8s-master ~]# kubectl delete pod mypod --grace-period=0 --force

# 过滤terminating的进行删除
[root@k8s-master ~]# kubectl get pod |grep Terminating|awk '{print $1}'|xargs kubectl delete pod --grace-period=0 --force

临时关闭daemonset

# 在调整ingress-controller的专属节点时，不想删除其配置信息，因为是ds无法设置副本数,只能将其的nodeselector配置到一个不存在的节点来临时关闭
[root@k8s-master ~]# kubectl patch daemonsets nginx-ingress-controller -p '{"spec":{"template":{"spec":{"nodeSelector":{"tier/ingress":"false"}}}}}

workload

重启workload

# 使用rollout restart来重启应用
[root@k8s-master ~]# kubectl -n <namespace> rollout restart deployment <deployment-name>

storageclass

给sc patch为默认sc

[root@k8s-master ~]# kubectl patch storageclass <your-class-name> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

正文完
k8s
2021-10-16
 
xadocker
版权声明：本站原创文章，由 xadocker 2021-10-16发表，共计12434字。
转载说明：除特殊说明外本站文章皆由CC-4.0协议发布，转载请注明出处。
在k8s中对存储做了一次性能测试
使用terraform+ansible交付个nacos集群
k8s prometheus-operator
k8s中的弹性伸缩
评论（没有评论）