Ingestion and querying with managed and self-deployed collection
Stay organized with collections
Save and categorize content based on your preferences.
This document describes how you might set up an environment that mixes
self-deployed collectors with managed collectors, across different
Google Cloud projects and clusters.
We strongly recommend using managed collection for all Kubernetes
environments; doing so practically eliminates the overhead of running
Prometheus collectors within your cluster.
You can run managed and self-deployed collectors within the same cluster.
We recommend using a consistent approach to monitoring, but you might choose
to mix deployment methods for some specific use cases, such as hosting a
push gateway, as illustrated in this document.
The following diagram illustrates a configuration that uses two
Google Cloud projects, three clusters, and mixes managed and self-deployed
collection. If you use only managed or self-deployed collection, then the
diagram is still applicable; just ignore the collection style you aren't using:
To set up and use a configuration like the one in the diagram, note the
following:
You must install any necessary exporters
in your clusters.
Google Cloud Managed Service for Prometheus does not install any exporters on your behalf.
Project 1 has a cluster running managed collection, which runs as a node
agent. Collectors are configured with
PodMonitoring resources to scrape
targets within a namespace and with
ClusterPodMonitoring resources
to scrape targets across a cluster. PodMonitorings must be applied in
every namespace in which you want to collect metrics.
ClusterPodMonitorings are applied once per cluster.
All data collected in Project 1 is saved in Monarch under
Project 1. This data is stored by default in the Google Cloud region
from which it emitted.
Project 2 has a cluster running self-deployed collection using
prometheus-operator and running as a standalone service. This cluster
is configured to use prometheus-operator PodMonitors or ServiceMonitors
to scrape exporters on pods or VMs.
Project 2 also hosts a push gateway sidecar to gather metrics from
ephemeral workloads.
All data collected in Project 2 is saved in Monarch under
Project 2. This data is stored by default in the Google Cloud region
from which it emitted.
Project 1 also has a cluster running Grafana and the
data source syncer. In this example, these
components are hosted in a standalone cluster, but they can be hosted in
any single cluster.
The data source syncer is configured to use scoping_project_A,
and its underlying service account has Monitoring Viewer
permissions for scoping_project_A.
When a user issues queries from Grafana,
Monarch expands scoping_project_A into its constituent
monitored projects and returns results for both Project 1 and Project 2
across all Google Cloud regions. All metrics retain their original
project_id and location (Google Cloud region) labels for
grouping and filtering purposes.
Do not federate when using Managed Service for Prometheus.
To reduce cardinality and cost by "rolling up" data before sending it to
Monarch, use local aggregation instead. For more information,
see Configure local aggregation.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-29 UTC."],[],[],null,["# Ingestion and querying with managed and self-deployed collection\n\nThis document describes how you might set up an environment that mixes\nself-deployed collectors with managed collectors, across different\nGoogle Cloud projects and clusters.\n\nWe strongly recommend using managed collection for all Kubernetes\nenvironments; doing so practically eliminates the overhead of running\nPrometheus collectors within your cluster.\nYou can run managed and self-deployed collectors within the same cluster.\nWe recommend using a consistent approach to monitoring, but you might choose\nto mix deployment methods for some specific use cases, such as hosting a\npush gateway, as illustrated in this document.\n\nThe following diagram illustrates a configuration that uses two\nGoogle Cloud projects, three clusters, and mixes managed and self-deployed\ncollection. If you use only managed or self-deployed collection, then the\ndiagram is still applicable; just ignore the collection style you aren't using:\n\nTo set up and use a configuration like the one in the diagram, note the\nfollowing:\n\n- You must install any necessary [exporters](https://prometheus.io/docs/instrumenting/exporters/)\n in your clusters.\n Google Cloud Managed Service for Prometheus does not install any exporters on your behalf.\n\n- Project 1 has a cluster running managed collection, which runs as a node\n agent. Collectors are configured with\n [PodMonitoring](https://github.com/GoogleCloudPlatform/prometheus-engine/blob/v0.15.3/doc/api.md#podmonitoring) resources to scrape\n targets within a namespace and with\n [ClusterPodMonitoring](https://github.com/GoogleCloudPlatform/prometheus-engine/blob/v0.15.3/doc/api.md#clusterpodmonitoring) resources\n to scrape targets across a cluster. PodMonitorings must be applied in\n every namespace in which you want to collect metrics.\n ClusterPodMonitorings are applied once per cluster.\n\n All data collected in Project 1 is saved in Monarch under\n Project 1. This data is stored by default in the Google Cloud region\n from which it emitted.\n- Project 2 has a cluster running self-deployed collection using\n prometheus-operator and running as a standalone service. This cluster\n is configured to use prometheus-operator PodMonitors or ServiceMonitors\n to scrape exporters on pods or VMs.\n\n Project 2 also hosts a push gateway sidecar to gather metrics from\n ephemeral workloads.\n\n All data collected in Project 2 is saved in Monarch under\n Project 2. This data is stored by default in the Google Cloud region\n from which it emitted.\n- Project 1 also has a cluster running [Grafana](/stackdriver/docs/managed-prometheus/query#ui-grafana) and the\n [data source syncer](/stackdriver/docs/managed-prometheus/query#grafana-oauth). In this example, these\n components are hosted in a standalone cluster, but they can be hosted in\n any single cluster.\n\n The data source syncer is configured to use scoping_project_A,\n and its underlying service account has [Monitoring Viewer](/monitoring/access-control#mon_roles_desc)\n permissions for scoping_project_A.\n\n When a user issues queries from Grafana,\n Monarch expands scoping_project_A into its constituent\n monitored projects and returns results for both Project 1 and Project 2\n across all Google Cloud regions. All metrics retain their original\n `project_id` and `location` (Google Cloud region) labels for\n grouping and filtering purposes.\n\nIf your cluster is not running inside Google Cloud, you must manually\nconfigure the `project_id` and `location` labels. For information about setting\nthese values, see [Run Managed Service for Prometheus outside of\nGoogle Cloud](/stackdriver/docs/managed-prometheus/setup-unmanaged#gmp-outside-gcp).\n\n**Do not federate when using Managed Service for Prometheus.**\nTo reduce cardinality and cost by \"rolling up\" data before sending it to\nMonarch, use local aggregation instead. For more information,\nsee [Configure local aggregation](/stackdriver/docs/managed-prometheus/cost-controls#local-aggregation)."]]
