DM-49296: Set SQLite constant_rows_max, and use it in join_data_coordinates.

[TallJimbo]

With QG generation now relying heavily on join_data_coordinates, we were hitting this limit sometimes.

Checklist

• ran Jenkins
• added a release note for user-visible changes to doc/changes
• (if changing dimensions.yaml) make a copy of dimensions.yaml in configs/old_dimensions

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.36%. Comparing base (31a8d53) to head (1ffefbe).
Report is 3 commits behind head on main.

✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1164   +/-   ##
=======================================
  Coverage   89.36%   89.36%           
=======================================
  Files         367      367           
  Lines       49546    49548    +2     
  Branches     6016     6016           
=======================================
+ Hits        44275    44277    +2     
  Misses       3852     3852           
  Partials     1419     1419           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[andy-slac]

Looks OK, one question.

[andy-slac]

The default one (in Database) is 100, does it mean Postgres limit is lower than SQLite?

[TallJimbo]

There's no limit for this specifically listed on the PostgreSQL limits doc page, so I suspect the de facto PostgreSQL limit one on total SELECT statement size or number of bind parameters. Keeping the limit really low here means we make temporary tables more often, which might be less efficient than we could be, but it guards pretty effectively against blowing one of those limits with really wide rows.

Or, in other words, "I don't think we know what we should set it to, but performance is okay with this."

[andy-slac]

I'm just surprised that you are reducing DirectQueryDriver._constant_rows_limit to 500 for SQLite and to 100 for Postgres. Before this change it was 1000 for both. My guess is that you hit 500 limit in SQLite tests, but does it also need much lower Postgres limit?

[TallJimbo]

Oh, that's a good point, when you put it that way, and all of my testing with PostgreSQL was with the 1000-row limit before I made this change. I think I will go change the default to that, as I think that's still plenty conservative enough.