Replace infinite scrolling with pagination in query tool data output …

…for better UX and performance. #1780

docs/en_US/preferences.rst

@@ -230,7 +230,7 @@ Use the fields on the *Options* panel to manage ERD preferences.
 
 
 * Use *Cardinality Notation* to change the cardinality notation format
-  used to present relationship links. 
+  used to present relationship links.
 
 * When the *SQL With DROP Table* switch is set to *True*, the SQL
   generated by the ERD Tool will add DROP table DDL before each CREATE
@@ -398,7 +398,7 @@ Use the fields on the *Editor* panel to change settings of the query editor.
   changed to text/plain. Keyword highlighting and code folding will be disabled.
   This will improve editor performance with large files.
 
-* When the *Highlight selection matches?* switch is set to *True*, the editor will 
+* When the *Highlight selection matches?* switch is set to *True*, the editor will
   highlight matched selected text.
 
 .. image:: images/preferences_sql_explain.png
@@ -476,11 +476,11 @@ Use the fields on the *Options* panel to manage editor preferences.
 * When the *Show View/Edit Data Promotion Warning?* switch is set to *True*
   View/Edit Data tool will show promote to Query tool confirm dialog on query edit.
 
-* When the *Underline query at cursor?* switch is set to *True*, query tool will 
+* When the *Underline query at cursor?* switch is set to *True*, query tool will
   parse and underline the query at the cursor position.
 
-* When the *Underlined query execute warning?* switch is set to *True*, query tool 
-  will warn upon clicking the *Execute Query* button in the query tool.  The warning 
+* When the *Underlined query execute warning?* switch is set to *True*, query tool
+  will warn upon clicking the *Execute Query* button in the query tool.  The warning
   will appear only if *Underline query at cursor?* is set to *False*.
 
 .. image:: images/preferences_sql_results_grid.png
@@ -497,9 +497,8 @@ preferences for copied data.
 * Specify the maximum width of the column in pixels when 'Columns sized by' is
   set to *Column data*. If 'Columns sized by' is set to *Column name* then this
   setting won't have any effect.
-* Specify the number of records to fetch in one batch in query tool when
-  query result set is large. Changing this value will override
-  ON_DEMAND_ROW_COUNT setting from config file.
+* Specify the number of records to fetch in one batch. Changing this value will
+  override DATA_RESULT_PAGE_SIZE setting from config file.
 * Use the *Result copy field separator* drop-down listbox to select the field
   separator for copied data.
 * Use the *Result copy quote character* drop-down listbox to select the quote
@@ -523,15 +522,15 @@ reformatting of SQL.
 
 * Use the *Data type case* option to specify whether to change data types
   into upper, lower, or preserve case.
-* Use the *Expression width* option to specify maximum number of characters 
+* Use the *Expression width* option to specify maximum number of characters
   in parenthesized expressions to be kept on single line.
 * Use the *Function case* option to specify whether to change function
   names into upper, lower, or preserve case.
 * Use the *Identifier case* option to specify whether to change identifiers
   (object names) into upper, lower, or capitalized case.
 * Use the *Keyword case* option to specify whether to change keywords into
   upper, lower, or preserve case.
-* Use *Lines between queries* to specify how many empty lines to leave 
+* Use *Lines between queries* to specify how many empty lines to leave
   between SQL statements. If set to zero it puts no new line.
 * Use *Logical operator new line* to specify newline placement before or
   after logical operators (AND, OR, XOR).

docs/en_US/query_tool.rst

@@ -74,8 +74,8 @@ key combination to select from a popup menu of autocomplete options.
 
 After entering a query, select the *Execute script* icon from the toolbar. The
 complete contents of the SQL editor panel will be sent to the database server
-for execution. To execute only a section of the code that is displayed in the 
-SQL editor, highlight the text that you want the server to execute, and click the 
+for execution. To execute only a section of the code that is displayed in the
+SQL editor, highlight the text that you want the server to execute, and click the
 *Execute script* icon.
 
 .. image:: images/query_execute_script.png
@@ -177,6 +177,7 @@ You can:
 * Use the *Save results to file* icon to save the content of the *Data Output*
   tab as a comma-delimited file.
 * Edit the data in the result set of a SELECT query if it is updatable.
+* Move between pages of data result.
 
 .. _updatable-result-set:
 

docs/en_US/query_tool_toolbar.rst

@@ -214,6 +214,37 @@ Data Editing Options
    | SQL                  | Use the SQL button to check the current query that gave the data.                                 |                |
    +----------------------+---------------------------------------------------------------------------------------------------+----------------+
 
+
+Pagination Options
+********************
+
+.. image:: images/query_data_pagination.png
+    :alt: Query tool data pagination options
+    :align: center
+
+.. table::
+   :class: longtable
+   :widths: 1 4 1
+
+   +----------------------+---------------------------------------------------------------------------------------------------+----------------+
+   | Icon                 | Behavior                                                                                          | Shortcut       |
+   +======================+===================================================================================================+================+
+   | *Page Range*         | Show the current row numbers visible in the data grid.                                            |                |
+   +----------------------+---------------------------------------------------------------------------------------------------+----------------+
+   | *Edit Page Range*    | Click to open the from and to page range inputs to allow setting them.                            |                |
+   +----------------------+---------------------------------------------------------------------------------------------------+----------------+
+   | *Page No*            | Enter the page no you want to jump to out of total shown next to this input                       |                |
+   +----------------------+---------------------------------------------------------------------------------------------------+----------------+
+   | *First Page*         | Click to go to the first page.                                                                    |                |
+   +----------------------+---------------------------------------------------------------------------------------------------+----------------+
+   | *Previous Page*      | Click to go to the previous page.                                                                 |                |
+   +----------------------+---------------------------------------------------------------------------------------------------+----------------+
+   | *Next Page*          | Click to go to the next page.                                                                     |                |
+   +----------------------+---------------------------------------------------------------------------------------------------+----------------+
+   | *Last Page*          | Click to go to the last page.                                                                     |                |
+   +----------------------+---------------------------------------------------------------------------------------------------+----------------+
+
+
 Status Bar
 **********
 

web/config.py

@@ -513,10 +513,10 @@
 SQLALCHEMY_TRACK_MODIFICATIONS = False
 
 ##########################################################################
-# Number of records to fetch in one batch in query tool when query result
-# set is large.
+# Number of records to fetch in one page in query tool when query result
+# set is large and is divided in multiple pages
 ##########################################################################
-ON_DEMAND_RECORD_COUNT = 1000
+DATA_RESULT_PAGE_SIZE = 1000
 
 ##########################################################################
 # Allow users to display Gravatar image for their username in Server mode

web/pgadmin/tools/sqleditor/__init__.py

@@ -15,6 +15,7 @@
 from urllib.parse import unquote
 from threading import Lock
 import threading
+import math
 
 import json
 from config import PG_DEFAULT_DRIVER, ALLOW_SAVE_PASSWORD, SHARED_STORAGE
@@ -106,8 +107,7 @@ def get_exposed_url_endpoints(self):
             'sqleditor.view_data_start',
             'sqleditor.query_tool_start',
             'sqleditor.poll',
-            'sqleditor.fetch',
-            'sqleditor.fetch_all',
+            'sqleditor.fetch_window',
             'sqleditor.fetch_all_from_start',
             'sqleditor.save',
             'sqleditor.inclusive_filter',
@@ -470,7 +470,8 @@ def _init_sqleditor(trans_id, connect, sgid, sid, did, dbname=None, **kwargs):
                             "prompt_password": True,
                             "allow_save_password": True
                             if ALLOW_SAVE_PASSWORD and
-                            session['allow_save_password'] else False,
+                            session.get('allow_save_password', None)
+                            else False,
                         }
                     ), '', ''
                 else:
@@ -925,7 +926,6 @@ def poll(trans_id):
     rows_affected = 0
     rows_fetched_from = 0
     rows_fetched_to = 0
-    has_more_rows = False
     columns = dict()
     columns_info = None
     primary_keys = None
@@ -936,8 +936,8 @@ def poll(trans_id):
     additional_messages = None
     notifies = None
     data_obj = {}
-    on_demand_record_count = Preferences.module(MODULE_NAME).\
-        preference('on_demand_record_count').get()
+    data_result_page_size = Preferences.module(MODULE_NAME).\
+        preference('data_result_page_size').get()
     # Check the transaction and connection status
     status, error_msg, conn, trans_obj, session_obj = \
         check_transaction_status(trans_id)
@@ -1004,7 +1004,7 @@ def poll(trans_id):
             status = 'Success'
             rows_affected = conn.rows_affected()
 
-            st, result = conn.async_fetchmany_2darray(on_demand_record_count)
+            st, result = conn.async_fetchmany_2darray(data_result_page_size)
 
             # There may be additional messages even if result is present
             # eg: Function can provide result as well as RAISE messages
@@ -1081,8 +1081,6 @@ def poll(trans_id):
                 # means nothing to fetch
                 if result and rows_affected > -1:
                     res_len = len(result)
-                    if res_len == on_demand_record_count:
-                        has_more_rows = True
 
                     if res_len > 0:
                         rows_fetched_from = trans_obj.get_fetched_row_cnt()
@@ -1126,6 +1124,15 @@ def poll(trans_id):
     data_obj['db_id'] = trans_obj.did \
         if trans_obj is not None and hasattr(trans_obj, 'did') else 0
 
+    page_size = rows_fetched_to - rows_fetched_from + 1
+    pagination = {
+        'page_size': page_size,
+        'page_count': math.ceil(conn.total_rows / page_size),
+        'page_no': math.floor(rows_fetched_from / page_size) + 1,
+        'rows_from': rows_fetched_from,
+        'rows_to': rows_fetched_to
+    }
+
     return make_json_response(
         data={
             'status': status, 'result': result,
@@ -1134,7 +1141,6 @@ def poll(trans_id):
             'rows_fetched_to': rows_fetched_to,
             'additional_messages': additional_messages,
             'notifies': notifies,
-            'has_more_rows': has_more_rows,
             'colinfo': columns_info,
             'primary_keys': primary_keys,
             'types': types,
@@ -1143,26 +1149,20 @@ def poll(trans_id):
             'oids': oids,
             'transaction_status': transaction_status,
             'data_obj': data_obj,
+            'pagination': pagination,
         }
     )
 
 
 @blueprint.route(
-    '/fetch/<int:trans_id>', methods=["GET"], endpoint='fetch'
-)
-@blueprint.route(
-    '/fetch/<int:trans_id>/<int:fetch_all>', methods=["GET"],
-    endpoint='fetch_all'
+    '/fetch_window/<int:trans_id>/<int:from_rownum>/<int:to_rownum>',
+    methods=["GET"], endpoint='fetch_window'
 )
 @pga_login_required
-def fetch(trans_id, fetch_all=None):
+def fetch_window(trans_id, from_rownum=0, to_rownum=0):
     result = None
-    has_more_rows = False
     rows_fetched_from = 0
     rows_fetched_to = 0
-    on_demand_record_count = Preferences.module(MODULE_NAME).preference(
-        'on_demand_record_count').get()
-    fetch_row_cnt = -1 if fetch_all == 1 else on_demand_record_count
 
     # Check the transaction and connection status
     status, error_msg, conn, trans_obj, session_obj = \
@@ -1174,33 +1174,39 @@ def fetch(trans_id, fetch_all=None):
                                   status=404)
 
     if status and conn is not None and session_obj is not None:
-        status, result = conn.async_fetchmany_2darray(fetch_row_cnt)
+        # rownums start from 0 but UI will ask from 1
+        status, result = conn.async_fetchmany_2darray(
+            records=None, from_rownum=from_rownum - 1, to_rownum=to_rownum - 1)
         if not status:
             status = 'Error'
         else:
             status = 'Success'
             res_len = len(result) if result else 0
-            if fetch_row_cnt != -1 and res_len == on_demand_record_count:
-                has_more_rows = True
 
             if res_len:
-                rows_fetched_from = trans_obj.get_fetched_row_cnt()
-                trans_obj.update_fetched_row_cnt(rows_fetched_from + res_len)
-                rows_fetched_from += 1
-                rows_fetched_to = trans_obj.get_fetched_row_cnt()
+                rows_fetched_from = from_rownum
+                rows_fetched_to = rows_fetched_from + res_len - 1
                 session_obj['command_obj'] = pickle.dumps(trans_obj, -1)
                 update_session_grid_transaction(trans_id, session_obj)
     else:
         status = 'NotConnected'
         result = error_msg
 
+    page_size = to_rownum - from_rownum + 1
+    pagination = {
+        'page_size': page_size,
+        'page_count': math.ceil(conn.total_rows / page_size),
+        'page_no': math.floor(rows_fetched_from / page_size) + 1,
+        'rows_from': rows_fetched_from,
+        'rows_to': rows_fetched_to
+    }
+
     return make_json_response(
         data={
             'status': status,
             'result': result,
-            'has_more_rows': has_more_rows,
-            'rows_fetched_from': rows_fetched_from,
-            'rows_fetched_to': rows_fetched_to
+            'pagination': pagination,
+            'row_count': conn.row_count,
         }
     )
 

web/pgadmin/tools/sqleditor/static/js/components/QueryToolConstants.js

@@ -37,7 +37,7 @@ export const QUERY_TOOL_EVENTS = {
   STOP_QUERY: 'STOP_QUERY',
   CURSOR_ACTIVITY: 'CURSOR_ACTIVITY',
   SET_MESSAGE: 'SET_MESSAGE',
-  ROWS_FETCHED: 'ROWS_FETCHED',
+  TOTAL_ROWS_COUNT: 'TOTAL_ROWS_COUNT',
   SELECTED_ROWS_COLS_CELL_CHANGED: 'SELECTED_ROWS_COLS_CELL_CHANGED',
   DATAGRID_CHANGED: 'DATAGRID_CHANGED',
   HIGHLIGHT_ERROR: 'HIGHLIGHT_ERROR',
@@ -56,8 +56,11 @@ export const QUERY_TOOL_EVENTS = {
   PUSH_HISTORY: 'PUSH_HISTORY',
   HANDLE_API_ERROR: 'HANDLE_API_ERROR',
   SET_FILTER_INFO: 'SET_FILTER_INFO',
-  FETCH_MORE_ROWS: 'FETCH_MORE_ROWS',
   REINIT_QT_CONNECTION:'REINIT_QT_CONNECTION',
+  FETCH_WINDOW: 'FETCH_WINDOW',
+  ALL_PAGE_ROWS_SELECTED: 'ALL_PAGE_ROWS_SELECTED',
+  ALL_ROWS_SELECTED: 'ALL_ROWS_SELECTED',
+  CLEAR_ROWS_SELECTED: 'CLEAR_ROWS_SELECTED',
 
   EDITOR_LAST_FOCUS: 'EDITOR_LAST_FOCUS',
   EDITOR_FIND_REPLACE: 'EDITOR_FIND_REPLACE',
@@ -109,4 +112,4 @@ export const PANELS = {
 
 export const MAX_QUERY_LENGTH = 1000000;
 
-export const OS_EOL = navigator.platform === 'win32' ? 'crlf' : 'lf';
+export const OS_EOL = navigator.platform === 'win32' ? 'crlf' : 'lf';