Understanding Ratelimit sampling

[manojm321]

I'm running 1.32 version of all components via docker image. I have setup remote sampling to 'ratelimiting'

          {
            "service_strategies": [
            ],
            "default_strategy": {
              "type": "ratelimiting",
              "param": 10.0,
              "operation_strategies": [
                {
                  "operation": "/health",
                  "type": "ratelimiting",
                  "param": 0.0
                },
                {
                  "operation": "/metrics",
                  "type": "ratelimiting",
                  "param": 0.0
                }
              ]
            }
          }

I see that agent presents this json doc

> curl http://localhost:5778/sampling?service=A | jq
{
  "strategyType": "RATE_LIMITING",
  "rateLimitingSampling": {
    "maxTracesPerSecond": 10
  },
  "operationSampling": {
    "defaultSamplingProbability": 0.001,
    "defaultLowerBoundTracesPerSecond": 0,
    "perOperationStrategies": null,
    "defaultUpperBoundTracesPerSecond": 0
  }
}

My understanding is by default any service will be able to send a max of 10traces/second. But in practice number of traces is way less. I believe this has to do with defaultSamplingProbability set to 0.001?

Also on jaeger UI I see the probability set to 0.001

What changes do I need to make so that I get 10traces/sec?

[yurishkuro]

I don't think you can combine rate limiting with per-operation sampling, as the latter only supports probabilistic sampling.

[manojm321]

Thanks, so ratelimiting is not supposed to be used as a default strategy?

What I'm trying to achieve with ratelimiting sampler is to work out a rate X(10traces/sec) with service owners for which they will get 100% sampling and guard the backed if the service becomes bursty(some other service DDOSing).

I believe the probabilistic sampler won't help in this case? Is the only option left to define each of the services in service_strategies":[] ?

[yurishkuro]

When you define per-operation section it takes precedence and it has its own default strategy which is always probabilistic.

You could look at adaptive sampling, it is expressed as a desired rates / sec, but then dynamically adjusts the sampling rates based on the observed traffic. Just know that it has a certain reaction time (2min by default iirc).