Skip to content

Commit

Permalink
Replace comma-delimited string flags with repeatable flags (prometheu…
Browse files Browse the repository at this point in the history 
…s-community#355)

* Replace comma-delimited string flags with repeatable flags

Signed-off-by: Tommy Li <[email protected]>

* preserve old flags but mark them as deprecated
Signed-off-by: Tommy Li <[email protected]>

---------

Signed-off-by: Tommy Li <[email protected]>
  • Loading branch information
@tommyzli
tommyzli authored Oct 7, 2024
1 parent 166886a commit 157cf40
Show file tree
Hide file tree
Showing 2 changed files with 60 additions and 27 deletions.
18 changes: 10 additions & 8 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -78,11 +78,11 @@ If you are still using the legacy [Access scopes][access-scopes], the `https://w

| Flag | Required | Default | Description |
| ----------------------------------- | -------- |---------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `google.project-id` | No | GCloud SDK auto-discovery | Comma seperated list of Google Project IDs |
| `google.project-ids` | No | GCloud SDK auto-discovery | Repeatable flag of Google Project IDs |
| `google.projects.filter` | No | | GCloud projects filter expression. See more [here](https://cloud.google.com/sdk/gcloud/reference/projects/list). |
| `monitoring.metrics-ingest-delay` | No | | Offsets metric collection by a delay appropriate for each metric type, e.g. because bigquery metrics are slow to appear |
| `monitoring.drop-delegated-projects` | No | No | Drop metrics from attached projects and fetch `project_id` only. |
| `monitoring.metrics-type-prefixes` | Yes | | Comma separated Google Stackdriver Monitoring Metric Type prefixes (see [example][metrics-prefix-example] and [available metrics][metrics-list]) |
| `monitoring.metrics-prefixes` | Yes | | Repeatable flag of Google Stackdriver Monitoring Metric Type prefixes (see [example][metrics-prefix-example] and [available metrics][metrics-list]) |
| `monitoring.metrics-interval` | No | `5m` | Metric's timestamp interval to request from the Google Stackdriver Monitoring Metrics API. Only the most recent data point is used |
| `monitoring.metrics-offset` | No | `0s` | Offset (into the past) for the metric's timestamp interval to request from the Google Stackdriver Monitoring Metrics API, to handle latency in published metrics |
| `monitoring.filters` | No | | Additonal filters to be sent on the Monitoring API call. Add multiple filters by providing this parameter multiple times. See [monitoring.filters](#using-filters) for more info. |
Expand Down Expand Up @@ -143,8 +143,9 @@ If we want to get all `CPU` (`compute.googleapis.com/instance/cpu`) and `Disk` (

```
stackdriver_exporter \
--google.project-id=my-test-project \
--monitoring.metrics-type-prefixes "compute.googleapis.com/instance/cpu,compute.googleapis.com/instance/disk"
--google.project-ids=my-test-project \
--monitoring.metrics-prefixes "compute.googleapis.com/instance/cpu"
--monitoring.metrics-prefixes "compute.googleapis.com/instance/disk"
```

### Using filters
Expand All @@ -163,16 +164,17 @@ Example: \
pubsub.googleapis.com/subscription/num_undelivered_messages (apply to only the specific subscription metric) \

The `filter_query` will be applied to a final metrics API query when querying for metric data. You can read more about the metric API filter options in GCPs documentation https://cloud.google.com/monitoring/api/v3/filters

The final query sent to the metrics API already includes filters for project and metric type. Each applicable `filter_query` will be appended to the query with an AND

Full example
```
stackdriver_exporter \
--google.project-id=my-test-project \
--monitoring.metrics-type-prefixes='pubsub.googleapis.com/subscription,compute.googleapis.com/instance/cpu' \
--google.project-ids=my-test-project \
--monitoring.metrics-prefixes='pubsub.googleapis.com/subscription' \
--monitoring.metrics-prefixes='compute.googleapis.com/instance/cpu' \
--monitoring.filters='pubsub.googleapis.com/subscription:resource.labels.subscription_id=monitoring.regex.full_match("us-west4.*my-team-subs.*")' \
--monitoring.filters='compute.googleapis.com/instance/cpu:resource.labels.instance=monitoring.regex.full_match("us-west4.*my-team-subs.*")'
--monitoring.filters='compute.googleapis.com/instance/cpu:resource.labels.instance=monitoring.regex.full_match("us-west4.*my-team-subs.*")'
```

Using projects filter:
Expand Down
69 changes: 50 additions & 19 deletions stackdriver_exporter.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -58,9 +58,13 @@ var (
).Default("/metrics").String()

projectID = kingpin.Flag(
"google.project-id", "Comma seperated list of Google Project IDs.",
"google.project-id", "DEPRECATED - Comma seperated list of Google Project IDs. Use 'google.project-ids' instead.",
).String()

projectIDs = kingpin.Flag(
"google.project-ids", "Repeatable flag of Google Project IDs",
).Strings()

projectsFilter = kingpin.Flag(
"google.projects.filter", "Google projects search filter.",
).String()
Expand All @@ -86,10 +90,13 @@ var (
).Default("503").Ints()

// Monitoring collector flags

monitoringMetricsTypePrefixes = kingpin.Flag(
"monitoring.metrics-type-prefixes", "Comma separated Google Stackdriver Monitoring Metric Type prefixes.",
).Required().String()
"monitoring.metrics-type-prefixes", "DEPRECATED - Comma separated Google Stackdriver Monitoring Metric Type prefixes. Use 'monitoring.metrics-prefixes' instead.",
).String()

monitoringMetricsPrefixes = kingpin.Flag(
"monitoring.metrics-prefixes", "Google Stackdriver Monitoring Metric Type prefixes. Repeat this flag to scrape multiple prefixes.",
).Strings()

monitoringMetricsInterval = kingpin.Flag(
"monitoring.metrics-interval", "Interval to request the Google Stackdriver Monitoring Metrics for. Only the most recent data point is used.",
Expand Down Expand Up @@ -269,16 +276,29 @@ func main() {
kingpin.Parse()

logger := promlog.New(promlogConfig)
if *projectID != "" {
level.Warn(logger).Log("msg", "The google.project-id flag is deprecated and will be replaced by google.project-ids.")
}
if *monitoringMetricsTypePrefixes != "" {
level.Warn(logger).Log("msg", "The monitoring.metrics-type-prefixes flag is deprecated and will be replaced by monitoring.metrics-prefix.")
}
if *monitoringMetricsTypePrefixes == "" && len(*monitoringMetricsPrefixes) == 0 {
level.Error(logger).Log("msg", "At least one GCP monitoring prefix is required.")
os.Exit(1)
}

ctx := context.Background()
if *projectID == "" && *projectsFilter == "" {
level.Info(logger).Log("msg", "Neither projectID nor projectsFilter was provided. Trying to discover it")
var discoveredProjectIDs []string

if len(*projectIDs) == 0 && *projectID == "" && *projectsFilter == "" {
level.Info(logger).Log("msg", "Neither projectIDs nor projectsFilter was provided. Trying to discover it")
var err error
projectID, err = getDefaultGCPProject(ctx)
defaultProject, err := getDefaultGCPProject(ctx)
if err != nil {
level.Error(logger).Log("msg", "no explicit projectID and error trying to discover default GCloud project", "err", err)
level.Error(logger).Log("msg", "no explicit projectIDs and error trying to discover default GCloud project", "err", err)
os.Exit(1)
}
discoveredProjectIDs = append(discoveredProjectIDs, *defaultProject)
}

monitoringService, err := createMonitoringService(ctx)
Expand All @@ -287,43 +307,54 @@ func main() {
os.Exit(1)
}

var projectIDs []string

if *projectsFilter != "" {
projectIDs, err = utils.GetProjectIDsFromFilter(ctx, *projectsFilter)
projectIDsFromFilter, err := utils.GetProjectIDsFromFilter(ctx, *projectsFilter)
if err != nil {
level.Error(logger).Log("msg", "failed to get project IDs from filter", "err", err)
os.Exit(1)
}
discoveredProjectIDs = append(discoveredProjectIDs, projectIDsFromFilter...)
}

if len(*projectIDs) > 0 {
discoveredProjectIDs = append(discoveredProjectIDs, *projectIDs...)
}
if *projectID != "" {
projectIDs = append(projectIDs, strings.Split(*projectID, ",")...)
discoveredProjectIDs = append(discoveredProjectIDs, strings.Split(*projectID, ",")...)
}

var metricsPrefixes []string
if len(*monitoringMetricsPrefixes) > 0 {
metricsPrefixes = append(metricsPrefixes, *monitoringMetricsPrefixes...)
}
if *monitoringMetricsTypePrefixes != "" {
metricsPrefixes = append(metricsPrefixes, strings.Split(*monitoringMetricsTypePrefixes, ",")...)
}

level.Info(logger).Log(
"msg", "Starting stackdriver_exporter",
"version", version.Info(),
"build_context", version.BuildContext(),
"projects", *projectID,
"metric_prefixes", *monitoringMetricsTypePrefixes,
"metric_prefixes", fmt.Sprintf("%v", metricsPrefixes),
"extra_filters", strings.Join(*monitoringMetricsExtraFilter, ","),
"projectIDs", fmt.Sprintf("%v", projectIDs),
"projectIDs", fmt.Sprintf("%v", discoveredProjectIDs),
"projectsFilter", *projectsFilter,
)

inputPrefixes := strings.Split(*monitoringMetricsTypePrefixes, ",")
metricsTypePrefixes := parseMetricTypePrefixes(inputPrefixes)
parsedMetricsPrefixes := parseMetricTypePrefixes(metricsPrefixes)
metricExtraFilters := parseMetricExtraFilters()
// drop duplicate projects
slices.Sort(discoveredProjectIDs)
uniqueProjectIds := slices.Compact(discoveredProjectIDs)

if *metricsPath == *stackdriverMetricsPath {
handler := newHandler(
projectIDs, metricsTypePrefixes, metricExtraFilters, monitoringService, logger, prometheus.DefaultGatherer)
uniqueProjectIds, parsedMetricsPrefixes, metricExtraFilters, monitoringService, logger, prometheus.DefaultGatherer)
http.Handle(*metricsPath, promhttp.InstrumentMetricHandler(prometheus.DefaultRegisterer, handler))
} else {
level.Info(logger).Log("msg", "Serving Stackdriver metrics at separate path", "path", *stackdriverMetricsPath)
handler := newHandler(
projectIDs, metricsTypePrefixes, metricExtraFilters, monitoringService, logger, nil)
uniqueProjectIds, parsedMetricsPrefixes, metricExtraFilters, monitoringService, logger, nil)
http.Handle(*stackdriverMetricsPath, promhttp.InstrumentMetricHandler(prometheus.DefaultRegisterer, handler))
http.Handle(*metricsPath, promhttp.Handler())
}
Expand Down

0 comments on commit 157cf40

Please sign in to comment.