Skip to content

Latest commit

 

History

History
158 lines (122 loc) · 4.65 KB

README.md

File metadata and controls

158 lines (122 loc) · 4.65 KB
Raw

Qubole Data Service Java SDK

A Java library that provides the tools you need to authenticate with, and use the Qubole Data Service API.

Installation

The SDK is available in The Central Repository. To use the SDK, add the following dependency to your Java application.

<dependency>
    <groupId>com.qubole.qds-sdk-java</groupId>
    <artifactId>qds-sdk-java</artifactId>
    <version>1.0.0</version>
</dependency>

NOTE: see the bullet below regarding Jersery 2.0

Usage

In your application initialization code, allocate a QdsClient object:

QdsConfiguration configuration = new DefaultQdsConfiguration(YOUR_API_KEY);
QdsClient client = QdsClientFactory.newClient(configuration);

YOUR_API_KEY referenced in above code is a placeholder for Qubole API Token. Please refer to following documentation for further details on how to obtain them: http://docs.qubole.com/en/latest/rest-api/api_overview.html#authentication

Then, make api calls as needed. E.g.

Future<CommandResponse> hiveCommandResponseFuture = client
    .command()
    .hive()
    .query("show tables;")
    .invoke();
CommandResponse hiveCommandResponse = hiveCommandResponseFuture.get();
...

Alternatively, you can use Jersey's callback mechanism. E.g.

InvocationCallback<CommandResponse> callback = new InvocationCallback<CommandResponse>()
{
    @Override
    public void completed(CommandResponse clusterItems)
    {
        // ...
    }

    @Override
    public void failed(Throwable throwable)
    {
        // ...
    }
};
client.command()
    .hive()
    .query("show tables;")
    .withCallback(callback)
    .invoke();
...

As part of your application's shutdown, close the client:

client.close();

Waiting for Results

Important: when you submit a command/query, it can take time for it to execute. You cannot get the result until it is ready. A utility is provided that polls the command and waits for the results: ResultLatch. You can use it to block in the foreground or using a callback.

Blocking:

ResultLatch latch = new ResultLatch(client, queryId);
ResultValue = latch.awaitResult();

With callback:

ResultLatch.Callback callback = new ResultLatch.Callback()
{
    @Override
    public void result(String queryId, ResultValue resultValue)
    {
        // use results
    }

    @Override
    public void error(String queryId, Exception e)
    {
        // handle error
    }
};
ResultLatch latch = new ResultLatch(client, queryId);
latch.callback(callback);

Streaming Results

Some Qubole APIs write large result sets to S3. If you would like to stream those results, use ResultStreamer. E.g.

ResultStreamer streamer = new ResultStreamer(client);  // save this until the end of your application

...

Future<ResultValue> results = client.command().results(id)...invoke();
Reader in = streamer.getResults(results.get());

As part of your application's shutdown, close the client:

streamer.close();

Paging

Some of the APIs support paging. These APIs have the "forPage" method. E.g.

// return page 2 using 3 per page
client.command().history().forPage(2, 3).invoke();

APIs

Using the QdsClient, you can access any of the Qubole APIs:

API Example
Reports client.report().allCommandsReport().start_date(...).end_date(...).limit(...).invoke();
Scheduler client.scheduler().list().invoke();
DbTaps client.dbTaps().list().invoke();
Hive Metadata client.hiveMetadata().getTableProperties("table").invoke();
Cluster client.cluster().list().invoke();
Command client.command().history().invoke();

Jersey 2.0

The SDK uses Jersey 2.0. Some widely used open source libraries such as Dropwizard are incompatible with Jersey 2.0. To workaround this incompatiblity, you can build the SDK using the Maven shade plugin which will hide the SDK's usage of Jersey 2.0. To build a shaded version of the SDK, follow these steps:

Javadoc

https://qubole.github.io/qds-sdk-java/apidocs/latest/