Bug - Scan activity & history table - error_message column too small

[psyray]

Issue Summary

While launching a scan the following error message is reported in the make logs when the message to log is larger than 300 chars.
Scan seems to crash without errors.

db_1           | 2023-09-08 09:28:44.379 UTC [22518] ERROR:  value too long for type character varying(300)
db_1           | 2023-09-08 09:28:44.379 UTC [22518] STATEMENT:  UPDATE "startScan_scanactivity" SET "status" = 0, "error_message" = 'HTTPSConnectionPool(host=''www.google.com'', port=443): Max retries exceeded with url: /search?q=(ext:pdf%20OR%20ext:doc%20OR%20ext:docx%20OR%20ext:xls%20OR%20ext:xlsx%20OR%20ext:ppt%20OR%20ext:pptx)+(site:*.xxxxx.com%20OR%20site:xxxxx.com)&num=50&start=0 (Caused by ProxyError(''Cannot connect to proxy.'', ConnectionResetError(104, ''Connection reset by peer'')))', "time" = '2023-09-08T09:28:44.378820+00:00'::timestamptz WHERE "startScan_scanactivity"."id" = 895
celery_1       | [2023-09-08 09:28:44,412: ERROR/ForkPoolWorker-189] Task reNgine.tasks.initiate_scan[5f80f0d3-1427-426a-94a9-ea3c95c72354] raised unexpected: DataError('value too long for type character varying(300)\n')

Full ST Scan activity

celery_1       | The above exception was the direct cause of the following exception:
celery_1       |
celery_1       | Traceback (most recent call last):
celery_1       |   File "/usr/local/lib/python3.9/dist-packages/celery/app/trace.py", line 451, in trace_task
celery_1       |     R = retval = fun(*args, **kwargs)
celery_1       |   File "/usr/local/lib/python3.9/dist-packages/celery/app/trace.py", line 734, in __protected_call__
celery_1       |     return self.run(*args, **kwargs)
celery_1       |   File "/usr/src/app/reNgine/tasks.py", line 356, in initiate_scan
celery_1       |     update_last_activity(activity_id, 0, error_message=str(e))
celery_1       |   File "/usr/src/app/reNgine/tasks.py", line 2107, in update_last_activity
celery_1       |     ScanActivity.objects.filter(
celery_1       |   File "/usr/local/lib/python3.9/dist-packages/django/db/models/query.py", line 783, in update
celery_1       |     rows = query.get_compiler(self.db).execute_sql(CURSOR)
celery_1       |   File "/usr/local/lib/python3.9/dist-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
celery_1       |     cursor = super().execute_sql(result_type)
celery_1       |   File "/usr/local/lib/python3.9/dist-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
celery_1       |     cursor.execute(sql, params)
celery_1       |   File "/usr/local/lib/python3.9/dist-packages/django/db/backends/utils.py", line 66, in execute
celery_1       |     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
celery_1       |   File "/usr/local/lib/python3.9/dist-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
celery_1       |     return executor(sql, params, many, context)
celery_1       |   File "/usr/local/lib/python3.9/dist-packages/django/db/backends/utils.py", line 84, in _execute
celery_1       |     return self.cursor.execute(sql, params)
celery_1       |   File "/usr/local/lib/python3.9/dist-packages/django/db/utils.py", line 90, in __exit__
celery_1       |     raise dj_exc_value.with_traceback(traceback) from exc_value
celery_1       |   File "/usr/local/lib/python3.9/dist-packages/django/db/backends/utils.py", line 84, in _execute
celery_1       |     return self.cursor.execute(sql, params)
celery_1       | django.db.utils.DataError: value too long for type character varying(300)

rengine/web/reNgine/tasks.py

Line 2102 in f5f3c22

def update_last_activity(id, activity_status, error_message=None):

ST History table

db_1           | 2023-09-08 10:18:44.737 UTC [23373] STATEMENT:  UPDATE "startScan_scanhistory" SET "start_scan_date" = '2023-09-08T10:14:29.390524+00:00'::timestamptz, "scan_status" = 1, "results_dir" = 'xxxxx.com_952762583585', "domain_id" = 2476, "scan_type_id" = 7, "celery_id" = 'd996449b-52a2-4b90-b016-9e459c0ca401', "subdomain_discovery" = true, "waf_detection" = true, "dir_file_fuzz" = false, "port_scan" = false, "fetch_url" = false, "vulnerability_scan" = false, "osint" = true, "screenshot" = true, "stop_scan_date" = NULL, "used_gf_patterns" = NULL, "error_message" = 'HTTPSConnectionPool(host=''www.google.com'', port=443): Max retries exceeded with url: /search?q=(ext:pdf%20OR%20ext:doc%20OR%20ext:docx%20OR%20ext:xls%20OR%20ext:xlsx%20OR%20ext:ppt%20OR%20ext:pptx)+(site:*.xxxxx.com%20OR%20site:xxxxx.com)&num=50&start=0 (Caused by ProxyError(''Cannot connect to proxy.'', OSError(''Tunnel connection failed: 404 Not Found'')))' WHERE "startScan_scanhistory"."id" = 117
celery_1       | [2023-09-08 10:18:44,745: ERROR/ForkPoolWorker-189] Task reNgine.tasks.initiate_scan[d996449b-52a2-4b90-b016-9e459c0ca401] raised unexpected: DataError('value too long for type character varying(300)\n')

Steps to Reproduce

1. Generate an error more than 300 chars

To resolve issue

Set error_message column up to 1000 chars in startScan_scanactivity & startScan_scanhistory table

[github-actions]

👋 Hi @psyray,
Issues is only for reporting a bug/feature request. Please read documentation before raising an issue https://rengine.wiki
For very limited support, questions, and discussions, please join reNgine Discord channel: https://discord.gg/azv6fzhNCE
Please include all the requested and relevant information when opening a bug report. Improper reports will be closed without any response.