kube-prometheus operator— Additional Scrape Configuration

Quickstart:

https://github.com/prometheus-operator/kube-prometheus

Apply additional scrape configuration

Create prometheus-additional.yaml:

$ cat prometheus-additional.yaml
- job_name: 'kubernetes-pods'
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: kubernetes_pod_name

Create secret:

kubectl create secret generic additional-scrape-configs \
--from-file=prometheus-additional.yaml \
--dry-run=client -oyaml > additional-scrape-configs.yaml

kubectl apply -f additional-scrape-configs.yaml -n monitoring

Update pod watch/list to prometheus-k8s ClusterRole:

- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - list
  - watch

Add this spec.additionalScrapeConfigs to Prometheus and Restart Prometheus statefulset:

spec:
  additionalScrapeConfigs:
    key: prometheus-additional.yaml
    name: additional-scrape-configs

Enable Prometheus to scrape a specific pod

spec:
  template:
    metadata:
      annotations:
        prometheus.io/port: "9121"
        prometheus.io/scrape: "true"

Verify:

Prometheus — Target health

That’s all

refer:

https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/additional-scrape-config.md

https://book.gimlet.io/docs/platform-components/observability/prometheus/