部署多套ingress-controller

1,885次阅读
没有评论

共计 7121 个字符，预计需要花费 18 分钟才能阅读完成。

部署多套ingress-controller

在k8s集群中一般用ingress来暴露服务到外部访问，如果项目过多且均需要被外部访问，单ingress-controller就负重前行扛大旗，所以我们需要对入口做流量的分流，避免相互影响，减少故障半径

nginx-ingress

查看官方文档，想要部署多套nginx-ingress则需要配置–ingress-class参数，默认为nginx

######## 略 
          args:
            - /nginx-ingress-controller
            - --publish-service=ingress-nginx/ingress-nginx-controller
            - --election-id=ingress-controller-leader

            # 多套nginx-ingress时，需要将此处配置不同值
            - --ingress-class=nginx
            - --configmap=ingress-nginx/ingress-nginx-controller
######## 略

配置该参数后，则在声明ingress资源时，需要指定ingress就可以使用以下方式

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
spec:
  ingressClassName: ingress1
######## 略 

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
spec:
  ingressClassName: ingress2
######## 略

部署多套nginx-ingress

此处我们使用helm部署nginx-ingress

为两套nginx-ingress分开namespace部署

[root@k8s-master ingress]# kubectl create ns ingress-frontend
namespace/ingress-frontend created
[root@k8s-master ingress]# kubectl create ns ingress-backend
namespace/ingress-backend created

# 配置ingress仓库
[root@k8s-master ingress]# helm repo add ingress https://kubernetes.github.io/ingress-nginx
[root@k8s-master ingress]# helm repo update

部署frontend专用ingress

[root@k8s-master ingress]# helm install ingress-frontend ingress/ingress-nginx --set controller.ingressClass=ingress-frontend --set controller.service.type=NodePort --set controller.service.httpPort.nodePort=30010 --set controller.stats.enabled=true --set controller.metrics.enabled=true --namespace ingress-frontend --version 3.41.0

部署backend专用ingress

[root@k8s-master ingress]# helm install ingress-backend ingress/ingress-nginx --set controller.ingressClass=ingress-backend --set controller.service.type=NodePort --set controller.service.httpPort.nodePort=30020 --set controller.stats.enabled=true --set controller.metrics.enabled=true --namespace ingress-backend --version 3.41.0

查看ingress-controller运行状态

[root@k8s-master ingress-prj]# kubectl get pod -n ingress-frontend
NAME                                                         READY   STATUS    RESTARTS   AGE
ingress-frontend-ingress-nginx-controller-676cdbf784-7fzzq   1/1     Running   0          14m
[root@k8s-master ingress-prj]# kubectl get pods -n ingress-backend
NAME                                                        READY   STATUS    RESTARTS   AGE
ingress-backend-ingress-nginx-controller-654dbc499b-9mqzd   1/1     Running   0          15m

测试使用不同nginx-ingress

部署demo服务

[root@k8s-master ingress-prj]# cat podinfo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: stefanprodan/podinfo
          ports:
            - containerPort: 9898
---
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  ports:
    - port: 80
      targetPort: 9898
  selector:
    app: api
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
        - name: frontend
          image: stefanprodan/podinfo
          ports:
            - containerPort: 9898
---
apiVersion: v1
kind: Service
metadata:
  name: frontend
spec:
  ports:
    - port: 80
      targetPort: 9898
  selector:
    app: frontend
  type: LoadBalancer

创建ingress资源

前端服务ingress资源

[root@k8s-master ingress-prj]# cat ingress-frontend.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: frontend
spec:
  ingressClassName: ingress-frontend
  rules:
    - host: "example.com"
      http:
        paths:
          - backend:
              serviceName: frontend
              servicePort: 80
            path: /
            pathType: Prefix

[root@k8s-master ingress-prj]# kubectl apply -f ingress-frontend.yaml
ingress.extensions/frontend created

后端服务ingress资源

[root@k8s-master ingress-prj]# cat ingress-backend.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: backend
spec:
  ingressClassName: ingress-backend
  rules:
    - host: "api.example.com"
      http:
        paths:
          - backend:
              serviceName: api
              servicePort: 80
            path: /
            pathType: Prefix

[root@k8s-master ingress-prj]# kubectl apply -f ingress-backend.yaml
ingress.extensions/backend created

查看ingress资源

[root@k8s-master ingress-prj]# kubectl get ingress
NAME       CLASS              HOSTS             ADDRESS   PORTS   AGE
backend    ingress-backend    api.example.com             80      116s
frontend   ingress-frontend   example.com                 80      7m21s

测试各自的流量入口

测试后端ingress入口

[root@k8s-master ~]# kubectl run -ti --rm=true busybox --image=busybox
# 测试后端ingress入口
/ # wget --header="Host: api.example.com" -qO- ingress-backend-ingress-nginx-controller.ingress-backend
{
  "hostname": "api-69f5868bb9-9tsqg",
  "version": "6.3.0",
  "revision": "e2e85a960447a56a1fa45747d2275abd28c13870",
  "color": "#34577c",
  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
  "message": "greetings from podinfo v6.3.0",
  "goos": "linux",
  "goarch": "amd64",
  "runtime": "go1.19.4",
  "num_goroutine": "6",
  "num_cpu": "8"
}/ #

# 测试后端ingress入口访问前端服务域名
/ #  wget --header="Host: example.com" -qO- ingress-backend-ingress-nginx-controller.ingress-backend
wget: server returned error: HTTP/1.1 404 Not Found

测试前端服务ingress入口

/ # wget --header="Host: example.com" --header="User-Agent: Mozilla" -qO- ingress-frontend-ingress-nginx-controller.ingress-frontend
<!DOCTYPE html>
<html>
<head>
  <title>frontend-85d76795d9-xzr2b</title>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
  <link rel="shortcut icon" type="image/png" href="https://kubernetes.io/images/favicon.png">
  <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400,500,700,900" rel="stylesheet">
  <link href="https://fonts.googleapis.com/css?family=Material+Icons" rel="stylesheet">
  <link href="https://cdn.jsdelivr.net/npm/vuetify@2.x/dist/vuetify.min.css" rel="stylesheet">
  <style>
        [v-cloak] {
    display: none;
  }
  .v-application .v-parallax {
    height: 100vh !important;
  }
  </style>
</head>
<body>
 <div id="app" v-cloak>
   <v-app dark>
    <v-content>
      <section>
        <v-parallax id="parallax-hero" :style="cuddleStyle" src="https://upload.wikimedia.org/wikipedia/commons/c/ca/1x1.png">
          <v-layout
            column
######## 略

# 测试前端ingress入口访问后端服务域名
/ # wget --header="Host: example.com" --header="User-Agent: Mozilla" -qO- ingress-frontend-ingress-nginx-controller.ingress-backend
wget: server returned error: HTTP/1.1 404 Not Found

通过指定ingressClassName: ingress-frontend从而实现不同ingress-controller入口，所以后续集群的入口可以按如下部署结构优化：

  • ingress-nginx使用hostport部署
  • ingress-nginx使用daemonset+nodeselector选择器部署
  • 按不同组织或项目划分多个ingress-controller
  • 每套ingress-controller都配有内外网slb
部署多套ingress-controller

prometheus监控ingress-nginx

ingress-nginx开启metrics功能

前面helm部署时已配置metrics开始，所以对应创建出来的pod和svc会有一个10254端口，该端口就是用来暴露ingress-nginx-controller服务的监控信息

[root@k8s-master ingress-prj]# kubectl get svc -n ingress-frontend
NAME                                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-frontend-ingress-nginx-controller             NodePort    10.96.89.103    <none>        80:30482/TCP,443:31314/TCP   154m
ingress-frontend-ingress-nginx-controller-admission   ClusterIP   10.96.124.62    <none>        443/TCP                      154m
ingress-frontend-ingress-nginx-controller-metrics     ClusterIP   10.96.250.201   <none>        10254/TCP                    154m

创建servicemonitor抓取ingress-nginx-controller

[root@k8s-master ingress-prj]# cat servicemonitor-ingress-frontend.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx-ingress-frontend
  namespace: monitoring
  labels:
    app.kubernetes.io/component: controller
spec:
  jobLabel: app.kubernetes.io/component
  endpoints:
  - port: metrics
    interval: 5s
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
  namespaceSelector:
    matchNames:
    - ingress-frontend

[root@k8s-master ingress-prj]# cat servicemonitor-ingress-backend.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx-ingress-backend
  namespace: monitoring
  labels:
    app.kubernetes.io/component: controller
spec:
  jobLabel: app.kubernetes.io/component
  endpoints:
  - port: metrics
    interval: 5s
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
  namespaceSelector:
    matchNames:
    - ingress-backend

查看targets

部署多套ingress-controller

查看grafana

部署多套ingress-controller

正文完
ingress k8s
2021-10-24
 
xadocker
版权声明：本站原创文章，由 xadocker 2021-10-24发表，共计7121字。
转载说明：除特殊说明外本站文章皆由CC-4.0协议发布，转载请注明出处。
使用Thanos来扩展prometheus的高可用
k8s-Service 控制器
k8s prometheus-operator
Prometheus-operator 对ingress的监控
评论（没有评论）