Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add method to upload sources from schema and metadata #82

Closed
wants to merge 6 commits into from
Closed

add method to upload sources from schema and metadata #82

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a4ea5d0
add ping test for crunchlake consumers
Alig1493 Apr 4, 2024
77e2c0d
add source post with schema and metadata params. add debug method for…
Alig1493 Apr 17, 2024
e550231
finalize payload and response for sources, schema and metadata upload
Alig1493 Apr 24, 2024
8d9e924
ammend add_schema_metadata to only accept required params. add docstr…
Alig1493 Apr 25, 2024
f7d0546
remove stray print. returning result as shoji.entity
Alig1493 Apr 25, 2024
01f2a2d
add json formatting
Alig1493 Apr 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions src/pycrunch/importing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,35 @@ def wait_for_batch_status(self, batch, status):
else:
raise ValueError("The batch did not reach the '%s' state in the "
"given time. Please check again later." % status)

def add_schema_metadata(self, site, schema, metadata, filename, fp, mimetype="application/x-parquet"):
"""
Create a new Source from a parquet file using schema and metadata.

Parameters:
site (shoji.Catalog): a shoji Catalog object, from which we acquire session and sources url
schema (dict): json string containing schema
metadata (dict): json string containing metadata
filename (str): name of file being uploaded
fp (BufferedReader): opened file object
mimetype (str): mimetype of file being uploaded

Returns:
shoji.Entity: Shoji entity containing the source url
"""
response = site.session.post(
site.catalogs.sources,
files={
"uploaded_file": (filename, fp, mimetype)
},
data={
"schema": json.dumps(schema),
"metadata": json.dumps(metadata),
"crunchlake": "create",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this argument?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's crunchlake specific. @mgdotdev can expand on this.

"dataset_id": "None"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no dataset ID to create a new source. Sources are not linked to datasets.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The argument is required for the sources post request endpoint. Otherwise there's errors.
See here: https://github.com/Crunch-io/zoom/blob/master/server/tests/controllers/test_sources.py#L282

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then it looks like that is incorrect. Sources are not linked to datasets.

This controller should not create sources depending on datasets. The relationship is the other way around. Sources do not point to datasets. Datasets point to sources.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm simply making a post request with the help of pycrunch to the endpoint that requires a dataset_id to be added. Otherwise I get the following error:

E pycrunch.lemonpy.ServerError: (<Response [500]>, 'https://test01.crunch.io/api/sources/', {'status': 500, 'exception': ['cr.lib.files.CrunchLakeMissingDatasetID: attr dataset_id: str is required for purposes of CrunchLake\n'], 'traceback': [['/var/lib/crunch.io/crserver_api_venv_python3.6/lib/python3.6/site-packages/cherrypy/_cprequest.py', 670, 'respond', 'response.body = self.handler()'], ['/var/lib/crunch.io/crserver_api_venv_python3.6/lib/python3.6/site-packages/cherrypy/lib/encoding.py', 221, '__call__', 'self.body = self.oldhandler(*args, **kwargs)'], ['/var/lib/crunch.io/crserver_api_venv_python3.6/lib/python3.6/site-packages/cr/core/monitoring/tracer.py', 903, 'inner', 'return f(*args, **kwargs)'], ['/var/lib/crunch.io/crserver_api_venv_python3.6/lib/python3.6/site-packages/cr/server/api/controllers/tools.py', 705, 'tracing_inner', 'return oldhandler(*args, **kwargs)'], ['/var/lib/crunch.io/crserver_api_venv_python3.6/lib/python3.6/site-packages/cr/server/shoji.py', 162, 'shojify_inner', 'result = request._pre_shojify_handler(*args, **kwargs)'], ['/var/lib/crunch.io/crserver_api_venv_python3.6/lib/python3.6/site-packages/cherrypy/_cpdispatch.py', 60, '__call__', 'return self.callable(*self.args, **self.kwargs)'], ['/var/lib/crunch.io/crserver_api_venv_python3.6/lib/python3.6/site-packages/cr/server/api/controllers/sources.py', 161, 'POST', '**options,'], ['/var/lib/crunch.io/crserver_api_venv_python3.6/lib/python3.6/site-packages/cr/core/monitoring/tracer.py', 903, 'inner', 'return f(*args, **kwargs)'], ['/var/lib/crunch.io/crserver_api_venv_python3.6/lib/python3.6/site-packages/cr/lib/entities/sources.py', 1049, 'from_file', 'creation_time=data["creation_time"],'], ['/var/lib/crunch.io/crserver_api_venv_python3.6/lib/python3.6/site-packages/cr/lib/files/__init__.py', 215, '__init__', '"attr dataset_id: str is required for purposes of CrunchLake"']], 'message': "['cr.lib.files.CrunchLakeMissingDatasetID: attr dataset_id: str is required for purposes of CrunchLake\\n']"})

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think this is a bug that has been introduced on the backend. And we should not grow this bug. Sources do not hold references to Datasets. Because multiple datasets are allowed to source rows from the same source entity. In which case, which of those multiple datasets does Source.dataset_id is pointing to?

@mgdotdev Why does the CrunchLake source require to know about a Dataset ID? For a fresh user, they will first have to upload a parquet file while they do not have any Datasets existing in the system yet. And only later they can create a Dataset by using this source. This relation is backwards.

}
)
return shoji.Entity(self=response.headers.get("Location"), session=site.session)

def add_source(self, ds, filename, fp, mimetype):
"""Create a new Source on the given dataset and return its URL."""
Expand Down
Loading