Prometheus Remote Write Integration

Introduction

The Prometheus Remote Write integration enables seamless metric data export from Prometheus to external systems at both partner and client levels. This integration ensures efficient data transfer, supporting advanced monitoring, analytics, and troubleshooting needs across diverse environments.

Prerequisites

An environment instrumented with Prometheus that has access to the OpsRamp portal.

Add a Prometheus Remote Write integration

Partner level

Select a Partner from the available list.
Navigate to Setup > Accounts.
Click on the Integrations tile.
The Installed Integrations page, where all the installed applications are displayed.
Note
If there are no installed applications, it will navigate to the Available Integrations page.
Click + ADD on the Installed Integrations page. The Available Integrations page displays all the available applications along with the newly created application with the version.
Note
You can even search for the application using the search option available. Also you can use the All Categories option to search.
Click +ADD in the Prometheus Remote Write application.
Enter a Name for the Prometheus Remote Write integration.
Select the Type: Normal or Containerized.
Normal: Select this type when Prometheus is installed on a virtual machine or bare metal machine.
Containerized: Select this type when Prometheus is installed in a container orchestration environment like Kubernetes.
Click Next.
For Normal deployment:
1. Follow the below given on-screen instructions displayed after installing the integration and click Finish:

Containerized

ConfigMap:

Follow the below given on-screen instructions displayed after installing the integration:

For Prometheus Operator:

Follow the below given on-screen instructions displayed after installing the integration:

Note

To deploy the Prometheus Remote Write integration to their clients, partners must include the tenant IDs in the configuration, ensuring the integration is installed across all selected clients.

Client level

Click All Clients and from the drop-down list, select a client.
Navigate to Setup > Accounts.
Click on the Integrations tile.
The Installed Integrations page, where all the installed applications are displayed.
Note
If there are no installed applications, it will navigate to the Available Integrations page.
Click + ADD on the Installed Integrations page. The Available Integrations page displays all the available applications along with the newly created application with the version.
Note
You can even search for the application using the search option available. Also you can use the All Categories option to search.
Click +ADD in the Prometheus Remote Write application.
Enter a Name for the Prometheus Remote Write integration.
Select the Type: Normal or Containerized.
Normal: Select this type when Prometheus is installed on a virtual machine or bare metal machine.
Containerized: Select this type when Prometheus is installed in a container orchestration environment like Kubernetes.
Click Next.
For Normal deployment:
1. Follow the below given on-screen instructions displayed after installing the integration and click Finish:

Containerized

ConfigMap:

Follow the below given on-screen instructions displayed after installing the integration:

For Prometheus Operator:

Follow the below given on-screen instructions displayed after installing the integration:

Sample configurations

Normal:

  global:
    scrape_interval: 15s
    evaluation_interval: 30s
    body_size_limit: 15MB
    sample_limit: 1500
    target_limit: 30
    label_limit: 30
    label_name_length_limit: 200
    label_value_length_limit: 200

    external_labels:
      OpsRampIntegrationName : DSF

  rule_files:
    - "first.rules"
    - "my/*.rules"

  remote_write:
    - url: 
      authorization:
        type: Bearer
        credentials: 
      headers:
        tenantId: 
        integrationUUID: 

  scrape_configs:
    - job_name: docker
      docker_sd_configs:
        - host: unix:///var/run/docker.sock

  alerting:
    alertmanagers:
      - scheme: https
        static_configs:
          - targets:
              - "1.2.3.4:9093"
              - "1.2.3.5:9093"
              - "1.2.3.6:9093"

Containerized:

ConfigMap:

global:
  scrape_interval: 15s
  evaluation_interval: 30s
  body_size_limit: 15MB
  sample_limit: 1500
  target_limit: 30
  label_limit: 30
  label_name_length_limit: 200
  label_value_length_limit: 200

  external_labels:
    OpsRampIntegrationName : DSF

rule_files:
  - "first.rules"
  - "my/*.rules"

remote_write:
  - url: 
    authorization:
      type: Bearer
      credentials: 
    headers:
      tenantId: 
      integrationUUID: 

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock

alerting:
  alertmanagers:
    - scheme: https
      static_configs:
        - targets:
            - "1.2.3.4:9093"
            - "1.2.3.5:9093"
            - "1.2.3.6:9093"

Operator:

Prometheus configuration YAML:

  apiVersion: monitoring.coreos.com/v1
  kind: Prometheus
  metadata:
    labels:
      app.kubernetes.io/component: prometheus
      app.kubernetes.io/instance: k8s
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/part-of: kube-prometheus
      app.kubernetes.io/version: 2.44.0
    name: k8s
    namespace: monitoring
  spec:
    alerting:
      alertmanagers:
      - apiVersion: v2
        name: alertmanager-main
        namespace: monitoring
        port: web
    enableFeatures: []
    externalLabels:
      OpsRampIntegrationName : gke-cluster
    image: quay.io/prometheus/prometheus:v2.44.0
    nodeSelector:
      kubernetes.io/os: linux
    podMetadata:
      labels:
        app.kubernetes.io/component: prometheus
        app.kubernetes.io/instance: k8s
        app.kubernetes.io/name: prometheus
        app.kubernetes.io/part-of: kube-prometheus
        app.kubernetes.io/version: 2.44.0
    podMonitorNamespaceSelector: {}
    podMonitorSelector: {}
    probeNamespaceSelector: {}
    probeSelector: {}
    replicas: 2
    resources:
      requests:
        memory: 400Mi
    ruleNamespaceSelector: {}
    ruleSelector: {}
    securityContext:
      fsGroup: 2000
      runAsNonRoot: true
      runAsUser: 1000
    serviceAccountName: prometheus-k8s
    serviceMonitorNamespaceSelector: {}
    serviceMonitorSelector: {}
    version: 2.44.0
    remoteWrite:
      - url: 
        authorization:
          type: 
          credentials:
            name:  # give name of the k8s secret 
            key: token # give the name of key from which the token needs to be taken
        headers:
          tenantId: 
          integrationUUID:

Prometheus Kubernetes Secret:

  apiVersion: v1
  kind: Secret
  metadata:
    name: 
  type: Opaque
  data:
    token:

Constraints

The number of active series per metric per client is 50000. You can avoid the limit by configuring Prometheus to filter metrics. For example, use the following configuration to limit apiserver_request_duration_seconds_bucket and etcd_request_duration_seconds_bucket metrics:

remoteWrite:
  - url: 
    authorization:
      type: Bearer
      credentials: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    headers:
      tenantId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      integrationUUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX
    writeRelabelConfigs:
    - action: drop
      regex: apiserver_request_duration_seconds_bucket
      sourceLabels:
      - __name__
    - action: drop
      regex: etcd_request_duration_seconds_bucket
      sourceLabels:
      - __name__

Configuration

Remote Write Tuning

Below are the recommended Prometheus Remote-Write Configuration tuning:

Configuration Parameter	Description
capacity	How many samples are queued in memory per shard before blocking reading from the WAL. When the WAL is blocked, samples cannot be appended to any shards and all throughput ceases.
max_shards	The maximum number of shards, or parallelism, Prometheus uses for each remote write queue.
min_shards	The minimum number of shards used by Prometheus, which is the number of shards used when remote write starts.
max_samples_per_send	The maximum number of samples sent per batch, which can be adjusted depending on the back end in use.
batch_send_deadline	The maximum time interval between sends for a single shard, in seconds.
min_backoff	The minimum time to wait before retrying a failed request, in seconds.
max_backoff	The maximum time to wait before retrying a failed request, in seconds.

See: https://prometheus.io/docs/practices/remote_write/ for more details.

Configuration examples to filter metrics pushed to remote write

Prometheus can be configured to filter the metrics to be pushed to a remote endpoint like the agent. See the Prometheus configuration remote_write property documentation for more information.

Drop samples example

The following writeRelabelConfigs configuration drops samples with a metric name that starts with go:

remote_write:
  - url: 
    authorization:
      type: Bearer
      credentials: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    headers:
      tenantId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      integrationUUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX
    writeRelabelConfigs:
    - sourceLabels: [__name__]
      regex: 'go_.*'
      action: drop

Keep samples example

The following writeRelabelConfigs configuration keeps samples with a metric name that starts with go:

remote_write:
  - url: 
    authorization:
      type: Bearer
      credentials: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    headers:
      tenantId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      integrationUUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX
    writeRelabelConfigs:
      - sourceLabels: [__name__]
        regex: 'go_.*'
        action: keep

Replace samples example

You can also change a label using the replace action before pushing it to a remote endpoint. The following example replaces samples with a metric name that starts with go with the name sample_go_label:

remote_write:
  - url: 
    authorization:
      type: Bearer
      credentials: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    headers:
      tenantId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      integrationUUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX
    writeRelabelConfigs:
    - sourceLabels: [__name__]
      regex: 'go_.*'
      action: replace
      targetLabels: sample_go_label