single-element rolling_std returns 0 (rather than null)

[MarcoGorelli]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

In [9]: df = pl.DataFrame({'ts': [datetime(2020, 1, 1)], 'val': [1]}).sort('ts')

In [10]: df.select(pl.col('val').rolling_std('1d', by='ts', closed='right'))
Out[10]:
shape: (1, 1)
┌─────┐
│ val │
│ --- │
│ f64 │
╞═════╡
│ 0.0 │
└─────┘

In [11]: df.select(pl.col('val').std())
Out[11]:
shape: (1, 1)
┌──────┐
│ val  │
│ ---  │
│ f64  │
╞══════╡
│ null │
└──────┘

Log output

No response

Issue description

The first example above returns 0, the second None

Expected behavior

Both should return None?

Noticed this while trying out #11134

Installed versions

INSTALLED VERSIONS
------------------
commit           : 0f437949513225922d851e9581723d82120684a6
python           : 3.10.6.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.10.102.1-microsoft-standard-WSL2
Version          : #1 SMP Wed Mar 2 00:30:59 UTC 2022
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_GB.UTF-8
LOCALE           : en_GB.UTF-8

pandas           : 2.0.3
numpy            : 1.25.1
pytz             : 2023.3
dateutil         : 2.8.2
setuptools       : 67.6.1
pip              : 23.1.2
Cython           : None
pytest           : 7.3.1
hypothesis       : 6.82.4
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 3.1.2
IPython          : 8.13.2
pandas_datareader: None
bs4              : 4.12.2
bottleneck       : None
brotli           : None
fastparquet      : 2023.7.0
fsspec           : 2023.6.0
gcsfs            : None
matplotlib       : 3.7.1
numba            : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : 12.0.1
pyreadstat       : None
pyxlsb           : None
s3fs             : None
scipy            : 1.11.0
snappy           : None
sqlalchemy       : 2.0.20
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
zstandard        : None
tzdata           : 2023.3
qtpy             : None
pyqt5            : None
None

[stinodego]

I believe this is related to #3717

Temporal windows just don't work correctly.

[MarcoGorelli]

not totally sure it's related to that one, I think the might just be a missing early return here - will take a look

Temporal windows just don't work correctly.

I'd like to think the default case with n>0 rows should be quite robust, there's extensive hypothesis tests for those - we should probably extend all that testing to cover the rest of the params too

---

note to future self: might need to look here:

polars/crates/polars-arrow/src/legacy/kernels/rolling/no_nulls/variance.rs

Lines 31 to 75 in 62f0929

	unsafe fn update(&mut self, start: usize, end: usize) -> T {
	// if we exceed the end, we have a completely new window
	// so we recompute
	let recompute_sum = if start >= self.last_end || self.last_recompute > 128 {
	self.last_recompute = 0;
	true
	} else {
	self.last_recompute += 1;
	// remove elements that should leave the window
	let mut recompute_sum = false;
	for idx in self.last_start..start {
	// safety
	// we are in bounds
	let leaving_value = self.slice.get_unchecked(idx);
	if T::is_float() && leaving_value.is_nan() {
	recompute_sum = true;
	break;
	}
	self.sum_of_squares -= *leaving_value * *leaving_value;
	}
	recompute_sum
	};
	self.last_start = start;
	// we traverse all values and compute
	if T::is_float() && recompute_sum {
	self.sum_of_squares = self
	.slice
	.get_unchecked(start..end)
	.iter()
	.map(|v| *v * *v)
	.sum::<T>();
	} else {
	for idx in self.last_end..end {
	let entering_value = *self.slice.get_unchecked(idx);
	self.sum_of_squares += entering_value * entering_value;
	}
	}
	self.last_end = end;
	self.sum_of_squares
	}
	}

[MarcoGorelli]

ok found it, looks like it's currently hard-coded to return 0 in this case:

polars/crates/polars-arrow/src/legacy/kernels/rolling/no_nulls/variance.rs

Lines 118 to 125 in 62f0929

	let denom = count - NumCast::from(self.ddof).unwrap();
	if end - start == 1 {
	T::zero()
	} else if denom <= T::zero() {
	//ddof would be greater than # of observations
	T::infinity()
	} else {
	let out = (sum_of_squares - count * mean * mean) / denom;

it'd be quite a refactor, but I think it probably should be

        if denom <= T::zero() {
            None
        }

and then update would return Option<T>

[MarcoGorelli]

Interesting, the same thing happens in group_by_rolling:

In [46]: df = pl.DataFrame({'ts': [datetime(2020, 1, 1), datetime(2020, 1, 2), datetime(2020, 1, 3)], 'val': [1, 2, 3]}
    ...: ).sort('ts')
    ...:
    ...: print(df.select(pl.col('val').rolling_std('1d', by='ts', closed='right')))
    ...:
    ...: print(df.group_by_rolling('ts', period='1s', closed='right').agg(pl.col('val'), val_std=pl.col('val').std()))
shape: (3, 1)
┌─────┐
│ val │
│ --- │
│ f64 │
╞═════╡
│ 0.0 │
│ 0.0 │
│ 0.0 │
└─────┘
shape: (3, 3)
┌─────────────────────┬───────────┬─────────┐
│ ts                  ┆ val       ┆ val_std │
│ ---                 ┆ ---       ┆ ---     │
│ datetime[μs]        ┆ list[i64] ┆ f64     │
╞═════════════════════╪═══════════╪═════════╡
│ 2020-01-01 00:00:00 ┆ [1]       ┆ 0.0     │
│ 2020-01-02 00:00:00 ┆ [2]       ┆ 0.0     │
│ 2020-01-03 00:00:00 ┆ [3]       ┆ 0.0     │
└─────────────────────┴───────────┴─────────┘

and there's tests that compare them, so they'll need fixing together