Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

data download can encounter too large an input in a database search. Probably should better validate this input too.... #406

Open
biblicabeebli opened this issue Oct 24, 2024 · 0 comments
Assignees
Labels
Bug Sounds like a bug!

Comments

@biblicabeebli
Copy link
Member

Data download can receive extremely large input when dealing with a large registry file.
Maybe we need to do this operation outside of the database query itself.
I'm better at optimizing queries now so I need to take a look at the performance of this query.
(and I need to modify data download to understand when the strleaming http request fails....)

OperationalError: sending query and params failed: number of parameters must be between 0 and 65535
  File "django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "psycopg/server_cursor.py", line 294, in execute
    raise ex.with_traceback(None)

OperationalError: sending query and params failed: number of parameters must be between 0 and 65535
  File "libs/streaming_zip.py", line 84, in __iter__
    for chunk, file_contents in chunks_and_content:
  File "multiprocessing/pool.py", line 868, in next
    raise value
  File "multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "multiprocessing/pool.py", line 144, in _helper_reraises_exception
    raise ex
  File "multiprocessing/pool.py", line 388, in _guarded_task_generation
    for i, x in enumerate(iterable):
  File "django/db/models/query.py", line 516, in _iterator
    yield from iterable
  File "django/db/models/query.py", line 208, in __iter__
    for row in compiler.results_iter(
  File "django/db/models/sql/compiler.py", line 1513, in results_iter
    results = self.execute_sql(
  File "django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)
  File "django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "psycopg/server_cursor.py", line 294, in execute
    raise ex.with_traceback(None)
@biblicabeebli biblicabeebli self-assigned this Oct 24, 2024
@biblicabeebli biblicabeebli added the Bug Sounds like a bug! label Oct 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Sounds like a bug!
Projects
None yet
Development

No branches or pull requests

1 participant