目录

kubernetes-k8s-prometheus

普罗米修斯也可监控k8s相关功能

相关仓库:https://github.com/coreos/kube-prometheus/

  1. 下载配置文件

    查看k8s版本 kubectl version,选择适配的版本

    kube-prometheus stack Kubernetes 1.29 Kubernetes 1.30 Kubernetes 1.31 Kubernetes 1.32 Kubernetes 1.33 Kubernetes 1.34
    release-0.14 ✔ ✔ ✔ x x x
    release-0.15 x x ✔ ✔ ✔ x
    release-0.16 x x ✔ ✔ ✔ ✔
    main x x x ✔ ✔ ✔

    git clone https://github.com/prometheus-operator/kube-prometheus.git -b release-0.16

    其他文件可删除保留 manifests 文件夹即可

  2. 下载相关镜像,导入到私有仓库内(以下以0.16版本为例)

    离线包以存入:k8s-images | OpenList

       quay.io/prometheus/alertmanager:v0.28.1
       quay.io/prometheus/blackbox-exporter:v0.27.0
       ghcr.io/jimmidyson/configmap-reload:v0.15.0
       quay.io/brancz/kube-rbac-proxy:v0.19.1
       grafana/grafana:12.1.0
       registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.16.0
       quay.io/prometheus/node-exporter:v1.9.1
       registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0
       quay.io/prometheus-operator/prometheus-operator:v0.85.0
       quay.io/prometheus/prometheus:v3.5.0
       quay.io/prometheus-operator/prometheus-config-reloader:v0.85.0
    
  3. 新建一个文件夹,将manifests 文件夹移动到里面

  4. 更改资源文件内的镜像地址,资源太多,推荐用下列脚本修改

       #!/bin/bash
       PRIVATE_REPO="hub.godone.top/pub/"   #这里修改为私有仓库地址
    
       # 第一步:通用 image 替换(保留镜像名)
       grep -rl "image: " *.yaml | while read FILENAME; do
         echo "Processing (generic image) $FILENAME"
         sed -i "s|image: .*/\([^/]*\):\([^[:space:]]*\)|image: ${PRIVATE_REPO}\1:\2|" "$FILENAME"
       done
       # 第二步:专门替换 prometheus-config-reloader 的完整路径(无论出现在哪)
       # 注意:这里直接在整个文件中搜索并替换,不限于 image 行
       for FILENAME in *.yaml; do
         #if grep -q "quay.io/prometheus-operator/prometheus-config-reloader" "$FILENAME"; then
         if grep -q "quay.io/prometheus-operator/prometheus-config-reloader" "$FILENAME"; then
           echo "Processing (specific reloader) $FILENAME"
           sed -i "s|quay.io/prometheus-operator/prometheus-config-reloader|${PRIVATE_REPO}prometheus-config-reloader|g" "$FILENAME"
         fi
       done
    
       echo "镜像地址替换完成"
    
  5. 部署资源文件

    部署资源:

       kubectl apply --server-side -f manifests/setup
       kubectl wait \
           --for condition=Established \
           --all CustomResourceDefinition \
           --namespace=monitoring
       kubectl apply -f manifests/
    

    删除资源:

       kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup
    
  6. 部署成功后会创建一个名为 monitoring 的命名空间,kube-prometheus相关资源都部署在这

       [root@k8s-master manifests]# kubectl -n monitoring get pod
       NAME                                   READY   STATUS    RESTARTS   AGE
       alertmanager-main-0                    2/2     Running   0          116m
       alertmanager-main-1                    2/2     Running   0          116m
       alertmanager-main-2                    2/2     Running   0          116m
       blackbox-exporter-79c7f8686b-xt7cm     3/3     Running   0          116m
       grafana-86c5d99cd4-r2dg7               1/1     Running   0          56m
       kube-state-metrics-5c5987cc47-pr2h8    3/3     Running   0          116m
       node-exporter-6vh47                    2/2     Running   0          116m
       node-exporter-8fvqn                    2/2     Running   0          116m
       node-exporter-wjfgs                    2/2     Running   0          116m
       prometheus-adapter-6f5f89cc6f-5df87    1/1     Running   0          116m
       prometheus-adapter-6f5f89cc6f-kmpqr    1/1     Running   0          116m
       prometheus-k8s-0                       2/2     Running   0          116m
       prometheus-k8s-1                       2/2     Running   0          116m
       prometheus-operator-77847d49f7-lzglb   2/2     Running   0          116m
    
  7. kube-prometheus默认只会创建ClusterIp,如果要在集群外访问的话,可以自己创建一个NodePode资源

       ---
       apiVersion: v1
       kind: Service
       metadata:
         labels:
           app.kubernetes.io/component: grafana
           app.kubernetes.io/name: grafana
           app.kubernetes.io/part-of: kube-prometheus
           app.kubernetes.io/version: 12.1.0
         name: grafana-NodePort
         namespace: monitoring
       spec:
         type: nodePort
         ports:
         - name: http
           port: 3000
           targetPort: 3000
       	nodePort: 30000
         selector:
           app.kubernetes.io/component: grafana
           app.kubernetes.io/name: grafana
           app.kubernetes.io/part-of: kube-prometheus
    
    

补充

有时候 grafana 可能会报以下错误

 Warning  Unhealthy  13m (x5 over 14m)    kubelet            Readiness probe failed: Get "http://10.244.1.222:3000/api/health": dial tcp 10.244.1.222:3000: connect: connection refused
 Warning  Unhealthy  10m (x3 over 13m)    kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503
 Warning  Unhealthy  4m9s (x40 over 13m)  kubelet            Readiness probe failed: Get "http://10.244.1.222:3000/api/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

这是因为grafana的健康检查所导致,可以修改manifests目录下的 grafana-deployment.yaml 配置文件,在以下位置加入新内容

...
        readinessProbe:
          httpGet:
            path: /api/health
            port: http
          initialDelaySeconds: 60   #容器启动后,等待 60 秒才开始第一次健康检查。
          periodSeconds: 30			#每隔 30 秒执行一次健康检查。
          timeoutSeconds: 10		#每次健康检查的超时时间为 10 秒。
          failureThreshold: 5		#连续 5 次健康检查失败后,才将 Pod 标记为“未就绪”。
        resources:
          limits:
            cpu: 200m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
...

推荐文章

Docker-网络模式
Docker-离线安装