stats::dds does not include sum

[abhie19]

Problem
It has been observed that when we are using the stats::sum(<event.value>) in the trickle script, restarting tremor produces a runtime error as follows:

2020-11-13T20:20:52.823615192+00:00 ERROR tremor_runtime::pipeline - error: Error(Script(BadAccessInEvent(Range(Location { unit_id: 0, line: 77, column: 22, absolute: 1605 }, Location { unit_id: 0, line: 77, column: 37, absolute: 1620 }), Range(Location { unit_id: 0, line: 77, column: 34, absolute: 1617 }, Location { unit_id: 0, line: 77, column: 37, absolute: 1620 }), "sum", ["min", "mean", "percentiles", "count", "max"])), State { next_error: None, backtrace: InternalBacktrace { backtrace: None } })

Below is the full trickle script to reproduce the bug-

#!config metrics_interval_s = 10

use std::type;

use std::record;

define script add_kafka_key

script
    use std::json;
    let $kafka_key = event.measurement+","+json::encode(event.tags);
    emit;
end;

create script add_kafka_key;

define tumbling window ten_secs
with
    interval = core::datetime::with_seconds(10),
script
    event.timestamp
end;

create stream normalize;

create stream aggregate;

define generic::batch operator batch

with

count = 3000,

timeout = 5

end;

create operator batch;

select {
"measurement": event.measurement,
"tags": event.tags,
"field": group[2],
"value": event.fields[group[2]],
"timestamp": event.timestamp,
}

from in
group by set(event.measurement, event.tags, each(record::keys(event.fields)))
into aggregate
having type::is_number(event.value);

select
{
"measurement": event.measurement,
"tags": event.tags,
"stats": aggr::stats::dds(event.value, [ "0.9", "0.95", "0.99" ]),
"field": event.field,
"timestamp": aggr::win::last(event.timestamp), # we can't use min since it's a float
}
from aggregate[`ten_secs`]
group by set(event.measurement, event.tags, event.field)
into normalize;

select {
"measurement": event.measurement,
"tags": event.tags,
"fields": {
"count_{event.field}": event.stats.count,
"min_{event.field}": event.stats.min,
"max_{event.field}": event.stats.max,
"mean_{event.field}": event.stats.mean,
"sum_{event.field}": event.stats.sum,
"p90_{event.field}": event.stats.percentiles["0.9"],
"p95_{event.field}": event.stats.percentiles["0.95"],
"p99_{event.field}": event.stats.percentiles["0.99"]
},
"timestamp": event.timestamp,
}

from normalize
where event.stats.count > 0
into add_kafka_key;

select event from add_kafka_key into batch;
select event from batch into out;

Steps

1. Update the config to include sum function
2. Save config
3. Restart tremor service
4. observe tremor logs for runtime errors

Possible Solution(s)

Notes

Output of rustup --version:
Output of rustup show:
Output of tremor --version:

[Licenser]

Hi @abhie19 thanks for reporting this!

the function stats::sum isn't called in the script, the error occurs when calling event.stats.sum. It seems that dds does not include this.

The quick solution, until we look into why dds doesn't includ the sum, is to use stats::sum() seperately and use that, we will take.

I also noticed that the error message is quite terrible and doesn't give the right information in the way we aim to provide it so I'll open a seperate ticket for that.

[mfelsche]

This has been fixed with #1246