Feature/upload multiple files at once to bytes #3476

Donnype · 2024-09-05T13:08:53Z

Changes

This makes the raw endpoint accept multiple files by default. Note that I think that utilizing the new functionality belongs in a subsequent PR, so there is no functionality added here and everything is backward compatible at the BytesClient level at least, mostly to keep PR's small and focused.

Issue link

Closes #3113

Demo

QA notes

These changes do not add functionality but affect Bytes, Boefjes and Rocky. Please verify that the boefjes still run properly, we can download raw files from Rocky, and we can still upload raw files from Rocky!

Code Checklist

All the commits in this PR are properly PGP-signed and verified.
This PR only contains functionality relevant to the issue.
I have written unit tests for the changes or fixes I made.
I have checked the documentation and made changes where necessary.
I have performed a self-review of my code and refactored it to the best of my abilities.
For any non-trivial functionality, I have added integration and/or end-to-end tests.

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

…ub.com:minvws/nl-kat-coordination into feature/upload-multiple-files-at-once-to-bytes

provided

Signed-off-by: Donny Peeters <[email protected]>

…s well Signed-off-by: Donny Peeters <[email protected]>

boefjes/boefjes/clients/bytes_client.py

Signed-off-by: Donny Peeters <[email protected]>

stephanie0x00 · 2024-09-09T13:06:28Z

During QA I ran into the following TypeError in the onboarding:

rocky-1  | {"event": "{\"code\": 500, \"request\": \"POST /en/onboarding/aa/step/setup-scan/URL/?report_type=dns-report\", \"exc_info\": true, \"event\": \"request_failed\", \"user_id\": 1, \"organization_code\": \"aa\", \"ip\": \"172.30.0.1\", \"request_id\": \"d246bc51-a5f6-49cf-9afe-701934dc8fdd\", \"level\": \"error\", \"timestamp\": \"2024-09-09T13:03:39.295245\"}", "exc_info": ["<class 'TypeError'>", "TypeError('Object of type set is not JSON serializable')", "<traceback object at 0x78795f514880>"]}
rocky-1  | Internal Server Error: /en/onboarding/aa/step/setup-scan/URL/
rocky-1  | Traceback (most recent call last):
rocky-1  |   File "/app/rocky/octopoes/connector/octopoes.py", line 50, in _verify_response
rocky-1  |     response.raise_for_status()
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 761, in raise_for_status
rocky-1  |     raise HTTPStatusError(message, request=request, response=self)
rocky-1  | httpx.HTTPStatusError: Client error '404 Not Found' for url 'http://octopoes_api/aa/object?reference=URL%7Cinternet%7Chttps%3A%2F%2Fmispo.es%2F&valid_time=2024-09-09%2013%3A03%3A38.803831%2B00%3A00'
rocky-1  | For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404
rocky-1  | 
rocky-1  | During handling of the above exception, another exception occurred:
rocky-1  | 
rocky-1  | Traceback (most recent call last):
rocky-1  |   File "/app/rocky/tools/ooi_helpers.py", line 247, in get_or_create_ooi
rocky-1  |     return api_connector.get(ooi.reference, observed_at), False
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/app/rocky/octopoes/connector/octopoes.py", line 96, in get
rocky-1  |     res = self.session.get(
rocky-1  |           ^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1054, in get
rocky-1  |     return self.request(
rocky-1  |            ^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 827, in request
rocky-1  |     return self.send(request, auth=auth, follow_redirects=follow_redirects)
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 914, in send
rocky-1  |     response = self._send_handling_auth(
rocky-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 942, in _send_handling_auth
rocky-1  |     response = self._send_handling_redirects(
rocky-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 999, in _send_handling_redirects
rocky-1  |     raise exc
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 982, in _send_handling_redirects
rocky-1  |     hook(response)
rocky-1  |   File "/app/rocky/octopoes/connector/octopoes.py", line 54, in _verify_response
rocky-1  |     raise ObjectNotFoundException(data["detail"])
rocky-1  | octopoes.models.exception.ObjectNotFoundException: URL|internet|https://mispo.es/
rocky-1  | 
rocky-1  | During handling of the above exception, another exception occurred:
rocky-1  | 
rocky-1  | Traceback (most recent call last):
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
rocky-1  |     response = get_response(request)
rocky-1  |                ^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
rocky-1  |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
rocky-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
rocky-1  |     return self.dispatch(request, *args, **kwargs)
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/mixins.py", line 109, in dispatch
rocky-1  |     return super().dispatch(request, *args, **kwargs)
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 143, in dispatch
rocky-1  |     return handler(request, *args, **kwargs)
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/edit.py", line 151, in post
rocky-1  |     return self.form_valid(form)
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/app/rocky/onboarding/views.py", line 149, in form_valid
rocky-1  |     ooi = self.get_or_create_url_object(cleaned_url)
rocky-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/app/rocky/onboarding/views.py", line 144, in get_or_create_url_object
rocky-1  |     url_ooi, _ = get_or_create_ooi(self.octopoes_api_connector, self.bytes_client, url, observed_at)
rocky-1  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/app/rocky/tools/ooi_helpers.py", line 249, in get_or_create_ooi
rocky-1  |     create_ooi(api_connector, bytes_client, ooi, observed_at, end_valid_time)
rocky-1  |   File "/app/rocky/tools/ooi_helpers.py", line 262, in create_ooi
rocky-1  |     bytes_client.add_manual_proof(task_id, BytesClient.raw_from_declarations([declaration]))
rocky-1  |   File "/app/rocky/rocky/bytes_client.py", line 58, in add_manual_proof
rocky-1  |     raw_id = self._save_raw(boefje_meta.id, raw, all_mime_types)
rocky-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/app/rocky/rocky/bytes_client.py", line 119, in _save_raw
rocky-1  |     response = self.session.post(
rocky-1  |                ^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1145, in post
rocky-1  |     return self.request(
rocky-1  |            ^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 814, in request
rocky-1  |     request = self.build_request(
rocky-1  |               ^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 357, in build_request
rocky-1  |     return Request(
rocky-1  |            ^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 340, in __init__
rocky-1  |     headers, stream = encode_request(
rocky-1  |                       ^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_content.py", line 212, in encode_request
rocky-1  |     return encode_json(json)
rocky-1  |            ^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_content.py", line 175, in encode_json
rocky-1  |     body = json_dumps(json).encode("utf-8")
rocky-1  |            ^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/json/__init__.py", line 231, in dumps
rocky-1  |     return _default_encoder.encode(obj)
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/json/encoder.py", line 200, in encode
rocky-1  |     chunks = self.iterencode(o, _one_shot=True)
rocky-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/json/encoder.py", line 258, in iterencode
rocky-1  |     return _iterencode(o, 0)
rocky-1  |            ^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/json/encoder.py", line 180, in default
rocky-1  |     raise TypeError(f'Object of type {o.__class__.__name__} '
rocky-1  | TypeError: Object of type set is not JSON serializable
rocky-1  | {"event": "Internal Server Error: /en/onboarding/aa/step/setup-scan/URL/", "exc_info": ["<class 'TypeError'>", "TypeError('Object of type set is not JSON serializable')", "<traceback object at 0x78795e1af800>"]}

Signed-off-by: Donny Peeters <[email protected]>

ammar92

Nice work! I've very little remarks that might improve it even more. There's still a little issue in the integration tests, but seems like a fixture mismatch. Should be fixable

bytes/bytes/api/router.py

bytes/bytes/models.py

bytes/bytes/api/router.py

…ue structure

stephanie0x00 · 2024-09-10T11:24:29Z

Checklist for QA:

I have checked out this branch, and successfully ran a fresh make reset.
I confirmed that there are no unintended functional regressions in this branch:
- I have managed to pass the onboarding flow
- Objects and Findings are created properly
- Tasks are created and completed properly
I confirmed that the PR's advertised feature or hotfix works as intended.
I checked the logs for errors and/or warnings and made issues where necessary

What works:

Looks good. Can generate and open reports, tasks and findings are created. Raw files can be uploaded (dnsrecords).

What doesn't work:

n/a

Bug or feature?:

n/a

Donnype added 9 commits August 28, 2024 08:02

WIP multipart upload

80edfcd

WIP multipart upload

ad59d77

Merge branch 'feature/upload-multiple-files-at-once-to-bytes' of gith…

8a18e39

…ub.com:minvws/nl-kat-coordination into feature/upload-multiple-files-at-once-to-bytes

WIP multipart upload

5758530

Merge branch 'feature/upload-multiple-files-at-once-to-bytes' of gith…

5d72fc0

…ub.com:minvws/nl-kat-coordination into feature/upload-multiple-files-at-once-to-bytes

Fix the test (client) and return raw ids based on the content type

edff901

provided

Update other clients as well

e7fcf3a

WIP: better safeguards for mime types

ff65977

Use boefjes API schema for the multiple raw file upload functionality

56730ae

Signed-off-by: Donny Peeters <[email protected]>

Donnype requested a review from a team as a code owner September 5, 2024 13:08

Donnype self-assigned this Sep 5, 2024

Donnype added the bytes Issues related to Bytes label Sep 5, 2024

Donnype added 4 commits September 5, 2024 15:12

Add comment about file_name usage

a785b56

Signed-off-by: Donny Peeters <[email protected]>

Fix integration test: order of mime-types change randomly

281b4bf

Signed-off-by: Donny Peeters <[email protected]>

Merge branch 'main' into feature/upload-multiple-files-at-once-to-bytes

0e1cd58

Fix integration tests by making the mime-types field a set in Bytes a…

2056796

…s well Signed-off-by: Donny Peeters <[email protected]>

underdarknl reviewed Sep 5, 2024

View reviewed changes

boefjes/boefjes/clients/bytes_client.py Show resolved Hide resolved

Donnype and others added 3 commits September 5, 2024 16:00

Fix integration tests part 2

16f2117

Signed-off-by: Donny Peeters <[email protected]>

Merge branch 'main' into feature/upload-multiple-files-at-once-to-bytes

f6060af

Merge branch 'main' into feature/upload-multiple-files-at-once-to-bytes

4c53b4c

Donnype and others added 2 commits September 9, 2024 15:12

A set is not JSON serializable

698053e

Signed-off-by: Donny Peeters <[email protected]>

Merge branch 'main' into feature/upload-multiple-files-at-once-to-bytes

7e250df

ammar92 reviewed Sep 10, 2024

View reviewed changes

Donnype and others added 4 commits September 10, 2024 12:06

Use httpx codes and the route signature to define the JSON return val…

e6bc372

…ue structure

Add integration test for uploading multiple raw files

bcd2de1

Fix integration test inconsistency with the order of the mime-types

7ad0fdb

Merge branch 'main' into feature/upload-multiple-files-at-once-to-bytes

d182714

underdarknl approved these changes Sep 10, 2024

View reviewed changes

underdarknl merged commit 60283ac into main Sep 10, 2024
20 checks passed

underdarknl deleted the feature/upload-multiple-files-at-once-to-bytes branch September 10, 2024 12:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/upload multiple files at once to bytes #3476

Feature/upload multiple files at once to bytes #3476

Donnype commented Sep 5, 2024

stephanie0x00 commented Sep 9, 2024

ammar92 left a comment

stephanie0x00 commented Sep 10, 2024

Feature/upload multiple files at once to bytes #3476

Feature/upload multiple files at once to bytes #3476

Conversation

Donnype commented Sep 5, 2024

Changes

Issue link

Demo

QA notes

Code Checklist

Checklist for code reviewers:

Checklist for QA:

stephanie0x00 commented Sep 9, 2024

ammar92 left a comment

Choose a reason for hiding this comment

stephanie0x00 commented Sep 10, 2024

Checklist for QA:

What works:

What doesn't work:

Bug or feature?: