Skip to content
This repository has been archived by the owner on Mar 26, 2024. It is now read-only.

Making it possible to push metrics in batches rather than one at a time #23

Merged
merged 1 commit into from
Jan 13, 2017

Conversation

wk8
Copy link
Contributor

@wk8 wk8 commented Jan 13, 2017

When using dogstatde to push a lot of custom Datadog metrics, we've
observed that the overhead of checking workers out & back in of the
dogstatsd_worker wpool can become a performance bottleneck.

This patch makes it possible to submit several metrics of the same type
at one time, hence allowing for a single worker check-out/in cycle to submit
them all, for example:

dogstatsd:gauge([{"users", UserTypeCount, #{ user_type => UserType }}
                 || {UserTypeCount, UserType} <- UserCounts]).

We've deployed that in production, and found a noticeable difference in our
use case.

This patch should be entirely backward-compatible.

Updated tests, and added a few more.

When using `dogstatde` to push a _lot_ of custom Datadog metrics, we've
observed that the overhead of checking workers out & back in of the
`dogstatsd_worker` `wpool` can become a performance bottleneck.

This patch makes it possible to submit several metrics of the same type
at one time, hence allowing for a single worker check-out/in cycle to submit
them all, for example:
```
dogstatsd:gauge([{"users", UserTypeCount, #{ user_type => UserType }}
                 || {UserTypeCount, UserType} <- UserCounts]).
```

We've deployed that in production, and found a noticeable difference in our
use case.

This patch should be entirely backward-compatible.

Updated tests, and added a few more.
@wk8
Copy link
Contributor Author

wk8 commented Jan 13, 2017

Hey @waisbrot long time no see ;)

@JoshRagem JoshRagem merged commit 1d83ac6 into WhoopInc:master Jan 13, 2017
@JoshRagem
Copy link

this looks great, thanks!

@wk8
Copy link
Contributor Author

wk8 commented Jan 13, 2017

@JoshRagem : thanks for quick merge, but found a stupid bug => #24 ... sorry!!

@waisbrot
Copy link
Contributor

@wk8 can you quantify "a lot"? I'm thinking that if it's far more than most people would be sending then it'd be nice to update the README so that users have an idea of when they need to worry about batching. And if it's a fairly low number we could look for ways to make batching happen automatically.

@wk8
Copy link
Contributor Author

wk8 commented Jan 13, 2017

@waisbrot we're seeing wpool starting to be a bottleneck at about 2 or 3 thousand metrics per minute, FWIW

@wk8
Copy link
Contributor Author

wk8 commented Jan 13, 2017

@waisbrot if you want => #25

@wk8
Copy link
Contributor Author

wk8 commented Jan 13, 2017

As to "look for ways to make batching happen automatically" => our own dogstatsd wrapper does just that, a simple gen_server that we push metrics to, and which then pushes those in batches to dogstatsd, either each 5 seconds or when the list grows bigger than 200, whichever happens first. Happy to add that to this repo, just wasn't sure if you guys would want that.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants