statsd-injector

The statsd injector is a colocated job for bosh VMs that transforms metrics from statsd format to loggregator envelope format, and sends them to the forwarder agent on the vm. It is being maintained but not actively developed.

If you have any questions, or want to get attention for a PR or issue please reach out on the #logging-and-metrics channel in the cloudfoundry slack

Usage

The statsd_injector job needs to be colocated with a Loggregator v2 envelope receiver on the loggregator_tls_statsdinjector.metron_port. It receives metrics via statsd UDP and re-emits them to the metric receiver.

Examples of loggregator v2 envelope receiver: loggregator forwarder agent.

A visual of how it fits in the broader loggregator architecture can be found in the Loggregator Architecture docs.

For example, this is used in CF by the components UAA and CAPI.

Development

The binary for statsd_injector is build from the code is src/. To run the tests:

cd src/
go test -mod=vendor ./... -race

Or, if you have ginkgo:

ginkgo -r -race -randomizeAllSpecs

Creating a release

This component runs as a bosh job. To build a local release:

bosh create-release

Deployment

1. Include a certificate variable in your bosh manifest:

   variables:
   +  - name: loggregator_tls_statsdinjector
   +    options:
   +      ca: loggregator_ca
   +      common_name: statsdinjector
   +      extended_key_usage:
   +      - client_auth
   +    type: certificate

2. Add the release to your deployment manifest.

   releases:
   +  - name: statsd-injector
   +    version: latest

   Then bosh upload-release the latest statsd-injector-release bosh release.

3. Colocate the job in the desired instance group.

   instance_groups:
   - name: <targeted_instance_group>
     jobs:
   +    - name: statsd_injector
   +      release: statsd-injector
   +      properties:
   +        loggregator:
   +          tls:
   +            ca_cert: "((loggregator_tls_statsdinjector.ca))"
   +            statsd_injector:
   +              cert: "((loggregator_tls_statsdinjector.certificate))"
   +              key: "((loggregator_tls_statsdinjector.private_key))"

   Then bosh deploy this updated manifest.

4. Send it a metric

   You can emit statsd metrics to the injector by sending a correctly formatted message to the udp port specified by the statsd_injector.statsd_port property. This defaults to port 8125 on the job's VM.

   As an example using nc:

   echo "origin.some.counter:1|c" | nc -u -w0 127.0.0.1 8125

   NOTE: The injector expects the the name of the metric to be of the form <origin>.<metric_name>

   The injector also supports tags according to the Datadog StatsD extension:

   echo "origin.some.counter:1|c|#testtag1:testvalue1,testtag2:testvalue2" | nc -u -w0 127.0.0.1 8125

5. Validate the metric can be seen.

   Assuming you are using statsd-injector with CF Release, you can use the CF Log Stream plugin

   cf log-stream <source-id> | grep <metric_name>

   Alternatively, you could curl the metrics-agent endpoint directly:

   curl https://localhost:14727/metrics -k -cacert=scrape_ca.crt --cert scrape.crt --key scrape.key

Pipeline

The Concourse pipeline configuration and set-up script live in the ci directory.

View the pipeline here.

To update the pipeline, you can run ci/set-pipeline.sh statsd-injector.

The pipeline has 4 groups:

• Claims an environment
• Run statsd-injector-release tests
  • Bumps Go modules and runs unit tests
  • Creates a deploys a dev release to CF-D
  • Tests against cfar-lats
  • Merges changes to release-elect and master branch when changes are accepted
• Cuts a release
• Bumps golang dependency