5.3.0

5.3.0 (2024-09-24)

• #664: Improvements to the AI SQL Assistant:

  • Table Annotations: Write persistent table annotations with descriptive information that will get injected into the
    prompt for the assistant. For example, if a table is commonly joined to another table through a non-obvious foreign
    key, you can tell the assistant about it in plain english, as an annotation to that table. Every time that table is
    deemed 'relevant' to an assistant request, that annotation will be included alongside the schema and sample data.
  • Few-Shot Examples: Using the small checkbox on the bottom-right of any saved queries, you can designate certain
    queries as 'few shot examples". When making an assistant request, any designated few-shot examples that reference
    the same tables as your assistant request will get included as 'reference sql' in the prompt for the LLM.
  • Autocomplete / multiselect when selecting tables info to send to the SQL Assistant. Much easier and more keyboard
    focused.
  • Relevant tables are added client-side visually, in real time, based on what's in the SQL editor and/or any tables
    mentioned in the assistant request. The dependency on sql_metadata is therefore removed, as server-side SQL parsing
    is no longer necessary.
  • Ability to view Assistant request/response history.
  • Improved system prompt that emphasizes the particular SQL dialect being used.
  • Addresses issue #657.

• #660: Userspace connection migration.

  • This should be an invisible change, but represents a significant refactor of how connections function. Instead of a
    weird blend of DatabaseConnection models and underlying Django models (which were the original Explorer
    connections), this migrates all connections to DatabaseConnection models and implements proper foreign keys to them
    on the Query and QueryLog models. A data migration creates new DatabaseConnection models based on the configured
    settings.EXPLORER_CONNECTIONS. Going forward, admins can create new Django-backed DatabaseConnection models by
    registering the connection in EXPLORER_CONNECTIONS, and then creating a DatabaseConnection model using the Django
    admin or the user-facing /connections/new/ form, and entering the Django DB alias and setting the connection type
    to "Django Connection".
  • The Query.connection and QueryLog.connection fields are deprecated and will be removed in a future release. They
    are kept around in this release in case there is an unforeseen issue with the migration. Preserving the fields for
    now ensures there is no data loss in the event that a rollback to an earlier version is required.

• Fixed a bug when validating connections to uploaded files. Also added basic locking when downloading files from S3.

• On-boarding UI; if no connections or queries are created, the UI walks the user through it a bit.

• Keyboard shortcut for formatting the SQL in the editor.

  • Cmd+Shift+F (Windows: Ctrl+Shift+F)
  • The format button has been moved tobe a small icon towards the bottom-right of the SQL editor.

• #675 - fail gracefully when building the schema if a particular table cant be accessed by the connection

5.3b2

5.3b1 (2024-09-10)

• #664: Improvements to the AI SQL Assistant:

  • Table Annotations: Write persistent table annotations with descriptive information that will get injected into the
    prompt for the assistant. For example, if a table is commonly joined to another table through a non-obvious foreign
    key, you can tell the assistant about it in plain english, as an annotation to that table. Every time that table is
    deemed 'relevant' to an assistant request, that annotation will be included alongside the schema and sample data.
  • Few-Shot Examples: Using the small checkbox on the bottom-right of any saved queries, you can designate certain
    queries as 'few shot examples". When making an assistant request, any designated few-shot examples that reference
    the same tables as your assistant request will get included as 'reference sql' in the prompt for the LLM.
  • Autocomplete / multiselect when selecting tables info to send to the SQL Assistant. Much easier and more keyboard
    focused.
  • Relevant tables are added client-side visually, in real time, based on what's in the SQL editor and/or any tables
    mentioned in the assistant request. The dependency on sql_metadata is therefore removed, as server-side SQL parsing
    is no longer necessary.
  • Ability to view Assistant request/response history.
  • Improved system prompt that emphasizes the particular SQL dialect being used.
  • Addresses issue #657.

• #660: Userspace connection migration.

  • This should be an invisible change, but represents a significant refactor of how connections function. Instead of a
    weird blend of DatabaseConnection models and underlying Django models (which were the original Explorer
    connections), this migrates all connections to DatabaseConnection models and implements proper foreign keys to them
    on the Query and QueryLog models. A data migration creates new DatabaseConnection models based on the configured
    settings.EXPLORER_CONNECTIONS. Going forward, admins can create new Django-backed DatabaseConnection models by
    registering the connection in EXPLORER_CONNECTIONS, and then creating a DatabaseConnection model using the Django
    admin or the user-facing /connections/new/ form, and entering the Django DB alias and setting the connection type
    to "Django Connection".
  • The Query.connection and QueryLog.connection fields are deprecated and will be removed in a future release. They
    are kept around in this release in case there is an unforeseen issue with the migration. Preserving the fields for
    now ensures there is no data loss in the event that a rollback to an earlier version is required.

• Fixed a bug when validating connections to uploaded files. Also added basic locking when downloading files from S3.

• On-boarding UI; if no connections or queries are created, the UI walks the user through it a bit.

• Keyboard shortcut for formatting the SQL in the editor.

  • Cmd+Shift+F (Windows: Ctrl+Shift+F)
  • The format button has been moved tobe a small icon towards the bottom-right of the SQL editor.

5.3b1

5.3b1 (2024-08-29)

• #664: Improvements to the AI SQL Assistant:

  • Table Annotations: Write persistent table annotations with descriptive information that will get injected into the
    prompt for the assistant. For example, if a table is commonly joined to another table through a non-obvious foreign
    key, you can tell the assistant about it in plain english, as an annotation to that table. Every time that table is
    deemed 'relevant' to an assistant request, that annotation will be included alongside the schema and sample data.
  • Few-Shot Examples: Using the small checkbox on the bottom-right of any saved queries, you can designate certain
    queries as 'few shot examples". When making an assistant request, any designated few-shot examples that reference
    the same tables as your assistant request will get included as 'reference sql' in the prompt for the LLM.
  • Autocomplete / multiselect when selecting tables info to send to the SQL Assistant. Much easier and more keyboard
    focused.
  • Relevant tables are added client-side visually, in real time, based on what's in the SQL editor and/or any tables
    mentioned in the assistant request. The dependency on sql_metadata is therefore removed, as server-side SQL parsing
    is no longer necessary.
  • Ability to view Assistant request/response history.
  • Improved system prompt that emphasizes the particular SQL dialect being used.
  • Addresses issue #657.

• #660: Userspace connection migration.

  • This should be an invisible change, but represents a significant refactor of how connections function. Instead of a
    weird blend of DatabaseConnection models and underlying Django models (which were the original Explorer
    connections), this migrates all connections to DatabaseConnection models and implements proper foreign keys to them
    on the Query and QueryLog models. A data migration creates new DatabaseConnection models based on the configured
    settings.EXPLORER_CONNECTIONS. Going forward, admins can create new Django-backed DatabaseConnection models by
    registering the connection in EXPLORER_CONNECTIONS, and then creating a DatabaseConnection model using the Django
    admin or the user-facing /connections/new/ form, and entering the Django DB alias and setting the connection type
    to "Django Connection".
  • The Query.connection and QueryLog.connection fields are deprecated and will be removed in a future release. They
    are kept around in this release in case there is an unforeseen issue with the migration. Preserving the fields for
    now ensures there is no data loss in the event that a rollback to an earlier version is required.

• Fixed a bug when validating connections to uploaded files. Also added basic locking when downloading files from S3.

• Keyboard shortcut for formatting the SQL in the editor.

  • Cmd+Shift+F (Windows: Ctrl+Shift+F)
  • The format button has been moved tobe a small icon towards the bottom-right of the SQL editor.

5.2.0

5.2.0 (2024-08-19)

• #651: Improvements to upload file abilities:

  • Ability to append an upload to a previously uploaded file/sqlite DB as a new table
  • Good cache busting and detection of file changes on uploads
  • Significant documentation improvements to uploads and connections
  • Separate the upload UI from the 'add connection' UI, as they are materially different
  • Ability to refresh a connection's schema and data (if it's an upload) from the connections list view

• #659: Search all queries, even if the header is collapsed. Addresses issue #464 (partially) and #658 (fully).

• #662: Refactored dockerfile to use non-root directories. Addresses issue #661.

• Fix a small bug with bar chart generation, when values are null

5.2b1

5.2b1 (2024-08-14)

• #651: Improvements to upload file abilities:

  • Ability to append an upload to a previously uploaded file/sqlite DB as a new table
  • Good cache busting and detection of file changes on uploads
  • Significant documentation improvements to uploads and connections
  • Separate the upload UI from the 'add connection' UI, as they are materially different
  • Ability to refresh a connection's schema and data (if it's an upload) from the connections list view

• #659: Search all queries, even if the header is collapsed. Addresses issue #464 (partially) and #658 (fully).

• #662: Refactored dockerfile to use non-root directories. Addresses issue #661.

• Fix a small bug with bar chart generation, when values are null

5.1.1

5.1.1 (2024-07-30)

• #654: Bugfix: Parameterized query does not work for viewers
• #653: Bugfix: Schema search not visible anymore
• #555: Prevent queries with many thousands of results from being punishingly slow. The number of data points in
  the chart now matches the number of data points in the preview pane.
• Bugfix: Error messages in Query detail view were floating in the wrong spot

5.1.0

5.1.0 (2024-07-30)

Major improvements:

• #647: Upload json files as data sources (in addition to CSV and SQLite files). Both 'normal'
  json files, and files structured as a list of json objects (one json object per line) are supported.
• #643: Addresses #640 (Snowflake support). Additionally, supports an "extras" field on the
  userspace DatabaseConnection object, which allows for arbitrary additional connection
  params to get added. This allows engine-specific (or just more obscure) settings to
  get injected into the connection.
• #644: Dockerfile and docker-compose to run the test_project. Replaces the old start.sh script.

Minor improvements:

• #647: In the schema explorer, clicking on a field name copies it to the clipboard
• #647: Charts are limited to a maximum of 10 series. This significantly speeds up rendering
  of 'wide' result-sets when charts are enabled.
• #645: Removed pie charts, added bar charts. Replaced Seaborn with Matplotlib
  because it's much lighter weight. Pie charts were overly finicky to get working.
  Bars are more useful. Will look to continue to expand charting in the future.
• #643: After uploading a csv/json/etc, the resulting connection is highlighted in the
  connection list, making it much clearer what happened.
• #643: Fixed some bugs in user connection stuff in general, and improved the UI.

Bugfixes and internal improvements:

• #647: Robustness to the user uploads feature, in terms of the UI, error handling and logging, and test coverage.
• #648: Backwards migration for 0016_alter_explorervalue_key.py
• #649: Use a more reliable source of the static files URL
• #635: Improved test coverage in tox, so that base requirements are properly used.
  This would have prevented (for example) issue 631. Additionally, introduced a test
  to verify that migrations are always generated, which would have prevented #633.
• #636: Output rendering bugfix.
• #567: Upgrade translate tags in templates to more modern style.

5.1b1

5.1.0b1 (2024-07-23)

Major improvements:

• #647: Upload json files as data sources (in addition to CSV and SQLite files). Both 'normal'
  json files, and files structured as a list of json objects (one json object per line) are supported.
• #643: Addresses #640 (Snowflake support). Additionally, supports an "extras" field on the
  userspace DatabaseConnection object, which allows for arbitrary additional connection
  params to get added. This allows engine-specific (or just more obscure) settings to
  get injected into the connection.
• #644: Dockerfile and docker-compose to run the test_project. Replaces the old start.sh script.

Minor improvements:

• #647: In the schema explorer, clicking on a field name copies it to the clipboard
• #647: Charts are limited to a maximum of 10 series. This significantly speeds up rendering
  of 'wide' result-sets when charts are enabled.
• #645: Removed pie charts, added bar charts. Replaced Seaborn with Matplotlib
  because it's much lighter weight. Pie charts were overly finicky to get working.
  Bars are more useful. Will look to continue to expand charting in the future.
• #643: After uploading a csv/json/etc, the resulting connection is highlighted in the
  connection list, making it much clearer what happened.
• #643: Fixed some bugs in user connection stuff in general, and improved the UI.

Bugfixes and internal improvements:

• #647: Robustness to the user uploads feature, in terms of the UI, error handling and logging, and test coverage.
• #648: Backwards migration for 0016_alter_explorervalue_key.py
• #649: Use a more reliable source of the static files URL
• #635: Improved test coverage in tox, so that base requirements are properly used.
  This would have prevented (for example) issue 631. Additionally, introduced a test
  to verify that migrations are always generated, which would have prevented #633.
• #636: Output rendering bugfix.
• #567: Upgrade translate tags in templates to more modern style.

5.0.2

5.0.2 (2024-07-3)

• #633 : Missing migration
• CSS tweaks to tighten up the Query UI

5.0.0

5.0.0 (2024-06-25)

• Manage DB connections via the UI (and/or Django Admin). Set EXPLORER_DB_CONNECTIONS_ENABLED
  to True in settings to enable user-facing connection management.
• Upload CSV or SQLite DBs directly, to create additional connections.
  This functionality has additional dependencies which can be installed with
  the 'uploads' extra (e.g. pip install django-sql-explorer[uploads]). Then set EXPLORER_USER_UPLOADS_ENABLED
  to True, and make sure S3_BUCKET is also set up.
• The above functionality is managed by a new license, restricting the
  ability of 3rd parties resell SQL Explorer (commercial usage is absolutely
  still permitted).
• Query List home page is sortable
• Select all / deselect all with AI assistant
• Assistant tests run reliably in CI/CD
• Introduced some branding and styling improvements