Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding missing script_location parameter while creating a Spark Command #99

Merged
merged 0 commits into from
Aug 7, 2019

Conversation

rohan-viz
Copy link

Closes #86

Although it is possible to submit a Spark command with a script location, this is not achievable through the SDK. This scenario is a very popular use case and these two are examples of important use case.

  1. When the program or script is long enough forming a request is painful using REST.
  2. A lot of times people already have scripts in s3 which are being used by other kinds of jobs. They always want to use the existing scripts and it becomes easy for them to just refer the path.

Rest API already supports this, refer here -
https://docs.qubole.com/en/latest/rest-api/command_api/submit-a-spark-command.html

@rohan-viz
Copy link
Author

closes #100
If I hit qubole API to create schedule directly with an invalid value of frequency say -1, I get the following response body.

{
"error": {
"error_code": 422,
"error_message": "Validation failed: Frequency must be an integer greater than 0"
}
}
But while using the qubole sdk,we get the response as javax.ws.rs.client.ResponseProcessingException: javax.ws.rs.ClientErrorException: HTTP 422 Unprocessable Entity

Hence somewhere you have lost the exact error message given by qubole API .
422 status code can be given by qubole for multiple reasons.

So in ErrorResponseFilter class, we are reading this error response from the input stream. Hence the client using the SDK will not be able to see the exact error message. It needs to be stick back to the response context.

@Sandy247 Sandy247 merged commit 509a8cb into qubole:master Aug 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

SparkCommandBuilder does not support scriptLocation
2 participants