Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase column limit to 1999 #50

Merged
merged 3 commits into from
Dec 30, 2024
Merged

Increase column limit to 1999 #50

merged 3 commits into from
Dec 30, 2024

Conversation

rkistner
Copy link
Contributor

We currently use JSON to store all synced data, with views to present it as a normal table.

Part of this is using triggers to persist updates (insert/update/delete), using the SQLite json_object function to convert the data back to JSON.

The issue is that SQLite limits function calls to 100 arguments by default, and this can be increased to a maximum of 127. Since json_object requires 2 arguments for each value (key + value), this means we can have a maximum of 50 or 63 columns, depending on the value of SQLITE_MAX_FUNCTION_ARG.

The change here works around the issue by building up the json object in chunks of 50 columns, then merge together using a new powersync_json_merge function. This takes the form of powersync_json_merge(json_object('a', 1, 'b', 2), json_object('c', 3, 'd', 4)). This is slightly less efficient than a single json_object call, so we only do this when there are more than 50 columns in a table.

With SQLITE_MAX_FUNCTION_ARG = 100, this would give us a maximum of 100 * 50 = 5000 columns. However, SQLITE_MAX_COLUMN with a default of 2000 is still a limiting factor, so we have a practical limit of 1999 columns (excluding id).

@rkistner rkistner marked this pull request as ready for review December 30, 2024 12:51
@rkistner rkistner merged commit 77f0917 into main Dec 30, 2024
12 checks passed
@rkistner rkistner deleted the remove-column-limits branch December 30, 2024 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants