Improve speed of `median` by implementing special `GroupsAccumulator` #13681

Rachelint · 2024-12-07T12:34:41Z

Which issue does this PR close?

Closes #13550

Rationale for this change

Support specific GroupsAccumulator for median for better performance.

What changes are included in this PR?

Impl MedianGroupsAccumulator
add related tests.

Are these changes tested?

Yes, by exist tests and new e2e and fuzzy tests.

Are there any user-facing changes?

No.

datafusion/functions-aggregate/src/median.rs

Rachelint · 2025-01-23T12:15:49Z

I am back and continue working this pr yesterday.
The rest work is fixing some failed fuzzy test cases and sorting out codes, plan to finish it today or tomorrow.

Very sorry for long delay for some private reason.

Thanks @Dandandan for helping!

Rachelint · 2025-01-25T11:04:46Z

I think this pr is ready now, sorry again for long delay.

Q6 in h2o medium:

result in main

Q6: SELECT id4, id5, MEDIAN(v3) AS median_v3, STDDEV(v3) AS sd_v3 FROM x GROUP BY id4, id5;
Query 6 iteration 1 took 9584.7 ms and returned 10000 rows
Query 6 iteration 2 took 9606.6 ms and returned 10000 rows
Query 6 iteration 3 took 9584.6 ms and returned 10000 rows

result in this pr

Q6: SELECT id4, id5, MEDIAN(v3) AS median_v3, STDDEV(v3) AS sd_v3 FROM x GROUP BY id4, id5;
Query 6 iteration 1 took 6558.1 ms and returned 10000 rows
Query 6 iteration 2 took 6473.9 ms and returned 10000 rows
Query 6 iteration 3 took 6494.3 ms and returned 10000 rows

2010YOUY01

Great! I have benchmarked locally and it's much faster (for benchmarks running on csvs, i think most time is spend reading from csv, so the results are closer)

h2o Q6 on parquet (10k groups):
main: 1500ms
pr: 300ms

query with 4 groups (from tpch sf10 lineitem table):
select median(l_orderkey) from lineitem group by l_returnflag, l_linestatus;
main: 0.7s
pr: 0.35s

I have a suggestion for testing: I noticed existing null tests for median() won't take this GroupsAccumulator path, those test cases don't have group by so they should be executed with regular Accumulator (see https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/aggregates/no_grouping.rs), could you include tests for null handling?

datafusion/datafusion/sqllogictest/test_files/aggregate.slt

Line 791 in f775791

# median with nulls

2010YOUY01 · 2025-01-26T09:13:37Z

datafusion/functions-aggregate/src/median.rs

+        // Extend values to related groups
+        // TODO: avoid using iterator of the `ListArray`, this will lead to
+        // many calls of `slice` of its ``inner array`, and `slice` is not
+        // so efficient(due to the calculation of `null_count` for each `slice`).


I think it's safe to directly use the value without checking null, null values should be ignored during accumulation

I think it's safe to directly use the value without checking null, null values should be ignored during accumulation

🤔 The input list is possible to be null actually, due to some of them are generated from convert_to_state(skip_partial).

And batch like:

row0: 0 row1: 1 row2: null ... rown: n

will be converted to a list like:

row0: [0] row1: [1] row2: null ... rown: [n]

I think we can implement a simple version for correctness firstly.

Rachelint · 2025-01-26T15:25:40Z

@2010YOUY01 testcases with nulls have been added.

2010YOUY01

The implementation looks good to me, thank you

Rachelint · 2025-01-27T13:54:50Z

Plan to merge this one tomorrow, would you like to review this pr again before merging?

@Dandandan @alamb

alamb

Thanks @Rachelint for this really nice PR and @2010YOUY01 for the review.

I thought this PR looks really really nice (easy to understand and read). I am running the end to end benchmarks on my gcp machine now to get final numbers but I suspect it will be much faster 🚀

I left various suggestions on ways to potentially make this PR faster, but they could all be done as follow ons (or never)

Thanks again 🙏

alamb · 2025-01-27T13:53:40Z

datafusion/core/tests/fuzz_cases/aggregate_fuzz.rs

+    let data_gen_config = baseline_config();
+
+    // Queries like SELECT median(a), median(distinct) FROM fuzz_table GROUP BY b
+    let query_builder = QueryBuilder::new()


alamb · 2025-01-27T14:06:00Z

datafusion/functions-aggregate/src/median.rs

+/// For calculating the accurate medians of groups, we need to store all values
+/// of groups before final evaluation.
+/// So values in each group will be stored in a `Vec<T>`, and the total group values
+/// will be actually organized as a `Vec<Vec<T>>`.


Given it is important to track the median values for each group separately I don't really see a way around Vec/Vec -- I think it is the simplest version and will have pretty reasonable performance

Yes, I tried not to use Vec<Vec<T>> for avoiding copying from Vec<Vec<T>> to the result Vec<T>, but it is hard to do that.

datafusion/functions-aggregate/src/median.rs

alamb · 2025-01-27T14:18:32Z

datafusion/functions-aggregate/src/median.rs

+
+        // `offsets` in `ListArray`, each row as a list element
+        let offsets = (0..=input_array.len() as i32).collect::<Vec<_>>();
+        let offsets = OffsetBuffer::new(ScalarBuffer::from(offsets));


likewise here OffsetBuffer::new_unchecked could be used

Done.
It is easy to ensure all check in OffsetBuffer::new can be passed by adding

assert!(input_array.len() <= i32::MAX as usize);

datafusion/functions-aggregate/src/median.rs

alamb

FWIW for completeness I also ran the sqllite suite against this PR too:

andrewlamb@Andrews-MacBook-Pro-2:~/Software/datafusion2$ INCLUDE_SQLITE=true cargo test --profile release-nonlto --test sqllogictests
...
andrewlamb@Andrews-MacBook-Pro-2:~/Software/datafusion2$ INCLUDE_SQLITE=true cargo test --profile release-nonlto --test sqllogictests
    Finished `release-nonlto` profile [optimized] target(s) in 0.34s
     Running bin/sqllogictests.rs (target/release-nonlto/deps/sqllogictests-6c6dc6221381c36b)
Completed in 8 minutes

And it all passed (thanks again to @Omega359 for making this happen)

I was trying to figure out how to run the extended test suite against this PR, but I couldn't figure out how to setup the workflow syntax. I filed this to track the idea:

Add a way to trigger the extended test suite from a PR #14319

alamb · 2025-01-27T21:34:14Z

BTW I am still trying to benchmark this branch to show how awesome it is. I am having trouble as the h2o large benchmark is being oomkilled on my machine

korowa · 2025-01-28T18:35:15Z

datafusion/functions-aggregate/src/median.rs

+#[derive(Debug)]
+struct MedianGroupsAccumulator<T: ArrowNumericType + Send> {
+    data_type: DataType,
+    group_values: Vec<Vec<T::Native>>,


Just wonder -- using Vec<Vec<>> for as a state storage doesn't seem to differ much from what regular accumulator does, but this PR still introduces a noticeable performance improvement. Are there any other optimizations that could be used in regular accumulator?

P.S. asking just because when I was doing +- same for count distinct (PR), the performance for GroupsAccumulator with Vec<HashSet<>> was not that significant comparing to regular accumulators with HashSet<> states.

I think among other things, the intermediate state management (creating ListArrays directly rather than from ScalarValue) probably helps a lot:

https://github.com/apache/datafusion/blob/6c9355d5be8b6045865fed67cb6d028b2dfc2e06/datafusion/functions-aggregate/src/median.rs#L200-L199

There is also an extra allocation per group when using the groups accumulator adapter thingie

That being said, it is a fair question how much better the existing MedianAccumulator could be if it built the ListArrays as does this PR directly 🤔

@korowa I think what mentioned by @alamb is a important point about the improvement.

Following are some other points for me:

in GroupsAccumulatorAdapter::update_batch, we need to reorder the input batch, and use slice to split the reordered batch after. I think such two operations may be not cheap.

datafusion/datafusion/functions-aggregate-common/src/aggregate/groups_accumulator.rs

Lines 241 to 265 in 6c9355d

let values = take_arrays(values, &batch_indices, None)?;

let opt_filter = get_filter_at_indices(opt_filter, &batch_indices)?;

// invoke each accumulator with the appropriate rows, first

// pulling the input arguments for this group into their own

// RecordBatch(es)

let iter = groups_with_rows.iter().zip(offsets.windows(2));

let mut sizes_pre = 0;

let mut sizes_post = 0;

for (&group_idx, offsets) in iter {

let state = &mut self.states[group_idx];

sizes_pre += state.size();

let values_to_accumulate = slice_and_maybe_filter(

&values,

opt_filter.as_ref().map(|f| f.as_boolean()),

offsets,

)?;

f(state.accumulator.as_mut(), &values_to_accumulate)?;

// clear out the state so they are empty for next

// iteration

state.indices.clear();

sizes_post += state.size();

in GroupsAccumulatorAdapter::merge_batch, the similar problem as input batch may be even more serious... Becasue we need to reorder a ListArray

and in GroupsAccumulatorAdapter::state, extra allocations exist as mentioned by @alamb .

@korowa it means impl a group accumulator for distinct count not get a obviously improvement?
It is really surprise for me, I am learning #8721

There was some improvements, but overall results for clickbench q9 (I was mostly looking at this query) were like x2.63 for GroupsAccumulator, and x2.30 for the regular Accumulator -- so it would be like 13-15% overall difference, which is not as massive as this PR results.

However, maybe things has changed in GroupsAccumulator implementation, and now even plain Vec<HashSet<>> will be way faster.

UPD: and, yes, maybe producing state, as pointed out by @alamb above, was (at least partially) the cause of non-significant improvement -- in count distinct it was implemented via ListArray::from_iter_primitive (commit), instead of building it from single flattened array and its offsets.

Seem really worth seeking the reason more deeply.

alamb · 2025-01-28T20:07:04Z

Sorry for the delay @Rachelint . I was having trouble with the benchmark queries

Here are my benchmark results -- not bad

almost 7x faster for our extended clickbench query:

datafusion/benchmarks/queries/clickbench/extended.sql

Line 5 in d051731

    
           SELECT "ClientIP", "WatchID",  COUNT(*) c, MIN("ResponseStartTiming") tmin, MEDIAN("ResponseStartTiming") tmed, MAX("ResponseStartTiming") tmax FROM hits WHERE "JavaEnable" = 0  GROUP BY  "ClientIP", "WatchID" HAVING c > 1 ORDER BY tmed DESC LIMIT 10;

And the actual h2o benchmark (which is dominated by CSV parsing) also shows a noticeable 1.6x improvement

datafusion/benchmarks/queries/h2o/groupby.sql

Line 6 in d051731

    
           SELECT id4, id5, MEDIAN(v3) AS median_v3, STDDEV(v3) AS sd_v3 FROM x GROUP BY id4, id5;

--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃  main_base ┃ impl-group-accumulator-for-medi… ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │  2753.78ms │                        2717.63ms │     no change │
│ QQuery 1     │   828.33ms │                         736.61ms │ +1.12x faster │
│ QQuery 2     │  1621.34ms │                        1507.86ms │ +1.08x faster │
│ QQuery 3     │   734.73ms │                         739.59ms │     no change │
│ QQuery 4     │ 12552.79ms │                        1823.32ms │ +6.88x faster │
│ QQuery 5     │ 19545.52ms │                       19039.51ms │     no change │
└──────────────┴────────────┴──────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (main_base)                           │ 38036.49ms │
│ Total Time (impl-group-accumulator-for-median)   │ 26564.53ms │
│ Average Time (main_base)                         │  6339.41ms │
│ Average Time (impl-group-accumulator-for-median) │  4427.42ms │
│ Queries Faster                                   │          3 │
│ Queries Slower                                   │          0 │
│ Queries with No Change                           │          3 │
└──────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark h2o.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃  main_base ┃ impl-group-accumulator-for-medi… ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │  2207.34ms │                        2168.72ms │     no change │
│ QQuery 2     │  5739.73ms │                        5757.47ms │     no change │
│ QQuery 3     │  4330.60ms │                        4332.62ms │     no change │
│ QQuery 4     │  3008.24ms │                        2998.72ms │     no change │
│ QQuery 5     │  4108.77ms │                        4072.95ms │     no change │
│ QQuery 6     │  6834.05ms │                        4160.14ms │ +1.64x faster │
│ QQuery 7     │  4059.20ms │                        4019.71ms │     no change │
│ QQuery 8     │  8013.99ms │                        8108.63ms │     no change │
│ QQuery 9     │ 10774.38ms │                       10642.69ms │     no change │
│ QQuery 10    │  8018.83ms │                        7916.58ms │     no change │
└──────────────┴────────────┴──────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (main_base)                           │ 57095.14ms │
│ Total Time (impl-group-accumulator-for-median)   │ 54178.23ms │
│ Average Time (main_base)                         │  5709.51ms │
│ Average Time (impl-group-accumulator-for-median) │  5417.82ms │
│ Queries Faster                                   │          1 │
│ Queries Slower                                   │          0 │
│ Queries with No Change                           │          9 │
└──────────────────────────────────────────────────┴────────────┘

Rachelint · 2025-01-29T04:21:05Z

@korowa would you like to review this one again before merging?

Rachelint · 2025-01-29T05:18:00Z

Will plan to merge this one tomorrow if there is not anyone else who would like time to review

korowa · 2025-01-29T05:30:24Z

datafusion/functions-aggregate/src/median.rs

+            .with_data_type(self.data_type.clone());
+
+        // `offsets` in `ListArray`, each row as a list element
+        assert!(input_array.len() <= i32::MAX as usize);


I wonder if we could use i32::try_from here instead of assert + following cast in range creation? I cannot imagine a real life use-case when this assertion will fail, but it still can be avoided

yes, i32::try_from is indeed better, fixed

korowa · 2025-01-29T05:32:26Z

@korowa would you like to review this one again before merging?

@Rachelint, I've partially went through it, and haven't found any major or blocking issues, so it looks like good to go.

Co-authored-by: Andrew Lamb <[email protected]>

korowa · 2025-01-29T07:20:01Z

datafusion/functions-aggregate/src/median.rs

+            .with_data_type(self.data_type.clone());
+
+        // `offsets` in `ListArray`, each row as a list element
+        let offset_end = i32::try_from(input_array.len()).unwrap();


And another one here -- why unwrap() and not ?, since we can return Error here?

Make sense, it is better to just use it to replace assert, fixed.
Thanks.

github-actions bot added the functions label Dec 7, 2024

Rachelint force-pushed the impl-group-accumulator-for-median branch from ded4e5f to 311f82f Compare December 8, 2024 17:28

Rachelint mentioned this pull request Dec 10, 2024

Sort out tests in aggregate.slt #13723

Open

github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Dec 21, 2024

Rachelint force-pushed the impl-group-accumulator-for-median branch from f80e890 to a53da8c Compare January 22, 2025 09:25

Dandandan reviewed Jan 23, 2025

View reviewed changes

datafusion/functions-aggregate/src/median.rs Outdated Show resolved Hide resolved

Dandandan mentioned this pull request Jan 23, 2025

Update ClickBench benchmarks with DataFusion 45.0.0 (When Published) #14246

Open

Dandandan reviewed Jan 23, 2025

View reviewed changes

datafusion/functions-aggregate/src/median.rs Outdated Show resolved Hide resolved

alamb mentioned this pull request Jan 23, 2025

Release DataFusion 45.0.0 #14008

Open

29 tasks

github-actions bot added the core Core DataFusion crate label Jan 24, 2025

Rachelint force-pushed the impl-group-accumulator-for-median branch from 5047371 to 40284c8 Compare January 24, 2025 09:13

Rachelint marked this pull request as ready for review January 25, 2025 11:04

2010YOUY01 reviewed Jan 26, 2025

View reviewed changes

2010YOUY01 approved these changes Jan 27, 2025

View reviewed changes

Dandandan approved these changes Jan 27, 2025

View reviewed changes

alamb approved these changes Jan 27, 2025

View reviewed changes

alamb mentioned this pull request Jan 27, 2025

Add a way to trigger the extended test suite from a PR #14319

Open

alamb changed the title ~~Support specific GroupsAccumulator for median~~ Improve speed of median by implementing special GroupsAccumulator Jan 27, 2025

alamb mentioned this pull request Jan 27, 2025

perf(array-agg): add fast path for array agg for merge_batch #14299

Merged

alamb reviewed Jan 27, 2025

View reviewed changes

Rachelint force-pushed the impl-group-accumulator-for-median branch 3 times, most recently from d2d8ca9 to a975adb Compare January 27, 2025 21:55

korowa reviewed Jan 28, 2025

View reviewed changes

korowa reviewed Jan 29, 2025

View reviewed changes

Rachelint force-pushed the impl-group-accumulator-for-median branch from abc0068 to 85ed001 Compare January 29, 2025 06:01

Rachelint and others added 20 commits January 29, 2025 14:15

draft of MedianGroupAccumulator.

cff822e

impl state.

7f10006

impl rest methods of MedianGroupsAccumulator.

6f172ef

improve comments.

cacc693

use MedianGroupsAccumulator.

11e6753

remove unused import.

955036f

add group_median_table to test group median.

17bd90b

complete group median test cases in aggregate slt.

28d8716

fix type of state.

1244df4

Clippy

c812350

Fmt

fdc9b33

add fuzzy tests for median.

e2f384f

fix decimal.

4b8a4ad

fix clippy.

5603bc0

improve comments.

7e6a73a

add median cases with nulls.

1c7b57a

Update datafusion/functions-aggregate/src/median.rs

5eb7711

Co-authored-by: Andrew Lamb <[email protected]>

use OffsetBuffer::new_unchecked in convert_to_state.

5a52e7c

add todo.

6f56a63

remove assert and switch to i32 try from.

e963d50

Rachelint force-pushed the impl-group-accumulator-for-median branch from 85ed001 to e963d50 Compare January 29, 2025 06:16

korowa reviewed Jan 29, 2025

View reviewed changes

return error when try from failed.

5fd9d8e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve speed of `median` by implementing special `GroupsAccumulator` #13681

Improve speed of `median` by implementing special `GroupsAccumulator` #13681

Rachelint commented Dec 7, 2024 •

edited

Loading

Rachelint commented Jan 23, 2025 •

edited

Loading

Rachelint commented Jan 25, 2025 •

edited

Loading

2010YOUY01 left a comment •

edited

Loading

2010YOUY01 Jan 26, 2025

Rachelint Jan 26, 2025 •

edited

Loading

Rachelint commented Jan 26, 2025

2010YOUY01 left a comment

Rachelint commented Jan 27, 2025

alamb left a comment

alamb Jan 27, 2025

alamb Jan 27, 2025

Rachelint Jan 27, 2025

alamb Jan 27, 2025

Rachelint Jan 27, 2025

alamb left a comment

alamb commented Jan 27, 2025

korowa Jan 28, 2025 •

edited

Loading

alamb Jan 28, 2025

Rachelint Jan 29, 2025 •

edited

Loading

Rachelint Jan 29, 2025

korowa Jan 29, 2025 •

edited

Loading

Rachelint Jan 29, 2025

alamb commented Jan 28, 2025

Rachelint commented Jan 29, 2025

Rachelint commented Jan 29, 2025 •

edited

Loading

korowa Jan 29, 2025

Rachelint Jan 29, 2025

korowa commented Jan 29, 2025

korowa Jan 29, 2025

Rachelint Jan 29, 2025 •

edited

Loading

	let values = take_arrays(values, &batch_indices, None)?;
	let opt_filter = get_filter_at_indices(opt_filter, &batch_indices)?;

	// invoke each accumulator with the appropriate rows, first
	// pulling the input arguments for this group into their own
	// RecordBatch(es)
	let iter = groups_with_rows.iter().zip(offsets.windows(2));

	let mut sizes_pre = 0;
	let mut sizes_post = 0;
	for (&group_idx, offsets) in iter {
	let state = &mut self.states[group_idx];
	sizes_pre += state.size();

	let values_to_accumulate = slice_and_maybe_filter(
	&values,
	opt_filter.as_ref().map(\|f\| f.as_boolean()),
	offsets,
	)?;
	f(state.accumulator.as_mut(), &values_to_accumulate)?;

	// clear out the state so they are empty for next
	// iteration
	state.indices.clear();
	sizes_post += state.size();

Improve speed of median by implementing special GroupsAccumulator #13681

Are you sure you want to change the base?

Improve speed of median by implementing special GroupsAccumulator #13681

Conversation

Rachelint commented Dec 7, 2024 • edited Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Rachelint commented Jan 23, 2025 • edited Loading

Rachelint commented Jan 25, 2025 • edited Loading

2010YOUY01 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rachelint Jan 26, 2025 • edited Loading

Choose a reason for hiding this comment

Rachelint commented Jan 26, 2025

2010YOUY01 left a comment

Choose a reason for hiding this comment

Rachelint commented Jan 27, 2025

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

alamb commented Jan 27, 2025

korowa Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rachelint Jan 29, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

korowa Jan 29, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb commented Jan 28, 2025

Rachelint commented Jan 29, 2025

Rachelint commented Jan 29, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

korowa commented Jan 29, 2025

Choose a reason for hiding this comment

Rachelint Jan 29, 2025 • edited Loading

Choose a reason for hiding this comment

Improve speed of `median` by implementing special `GroupsAccumulator` #13681

Improve speed of `median` by implementing special `GroupsAccumulator` #13681

Rachelint commented Dec 7, 2024 •

edited

Loading

Rachelint commented Jan 23, 2025 •

edited

Loading

Rachelint commented Jan 25, 2025 •

edited

Loading

2010YOUY01 left a comment •

edited

Loading

Rachelint Jan 26, 2025 •

edited

Loading

korowa Jan 28, 2025 •

edited

Loading

Rachelint Jan 29, 2025 •

edited

Loading

korowa Jan 29, 2025 •

edited

Loading

Rachelint commented Jan 29, 2025 •

edited

Loading

Rachelint Jan 29, 2025 •

edited

Loading