Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Getting Permission Denied and StreamWriterClosedException in BQ Storage Write Api #1882

Closed
SubhamSinghal opened this issue Nov 14, 2022 · 1 comment
Closed

Getting Permission Denied and StreamWriterClosedException in BQ Storage Write Api #1882

SubhamSinghal opened this issue Nov 14, 2022 · 1 comment
Labels
api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. status: duplicate Duplicate.

Comments

@SubhamSinghal
Copy link

I have been using java service which write records to BQ using storage api. Its been running fine for almost a week and then suddenly i see below mentioned error in application log.

java.lang.RuntimeException: com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: CheckToken failed: Proxy role cloud-dataengine-globalrouting is not trusted to perform delegation for proxied role cloudpath-cfe-prod.  Please check the flags --rpc_proxy_allowlist and --rpc_proxy_denylist, and/or the RPC_ServiceSecurityOptions supplied to this service.
	at bq.DataWriter$AppendCompleteCallback.onFailure(DataWriter.java:123)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:67)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
	at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:94)
	at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:76)
	at com.google.api.core.SettableApiFuture.setException(SettableApiFuture.java:51)
	at com.google.cloud.bigquery.storage.v1.ConnectionWorker.cleanupInflightRequests(ConnectionWorker.java:587)
	at com.google.cloud.bigquery.storage.v1.ConnectionWorker.appendLoop(ConnectionWorker.java:515)
	at com.google.cloud.bigquery.storage.v1.ConnectionWorker.access$000(ConnectionWorker.java:58)
	at com.google.cloud.bigquery.storage.v1.ConnectionWorker$1.run(ConnectionWorker.java:233)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: CheckToken failed: Proxy role cloud-dataengine-globalrouting is not trusted to perform delegation for proxied role cloudpath-cfe-prod.  Please check the flags --rpc_proxy_allowlist and --rpc_proxy_denylist, and/or the RPC_ServiceSecurityOptions supplied to this service.
	at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:98)
	at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:41)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:86)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
	at com.google.api.gax.grpc.ExceptionResponseObserver.onErrorImpl(ExceptionResponseObserver.java:82)
	at com.google.api.gax.rpc.StateCheckingResponseObserver.onError(StateCheckingResponseObserver.java:84)
	at com.google.api.gax.grpc.GrpcDirectStreamController$ResponseObserverAdapter.onClose(GrpcDirectStreamController.java:149)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:535)
	at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:470)
	at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:434)
	at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:467)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	... 1 more
Caused by: io.grpc.StatusRuntimeException: PERMISSION_DENIED: CheckToken failed: Proxy role cloud-dataengine-globalrouting is not trusted to perform delegation for proxied role cloudpath-cfe-prod.  Please check the flags --rpc_proxy_allowlist and --rpc_proxy_denylist, and/or the RPC_ServiceSecurityOptions supplied to this service.
	at io.grpc.Status.asRuntimeException(Status.java:535)
	... 17 more

and

java.lang.RuntimeException: com.google.cloud.bigquery.storage.v1.Exceptions$StreamWriterClosedException: FAILED_PRECONDITION: Connection is closed due to com.google.api.gax.rpc.InternalException: io.grpc.StatusRuntimeException: INTERNAL: A retriable error could not be retried due to Extensible Stubs memory limits for streams (see go/xs-retries-memory-limit-reached) (old status: RPC::STREAM_BROKEN: Connection to server broken (OnChannelError))
	at bq.DataWriter$AppendCompleteCallback.onFailure(DataWriter.java:123)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:67)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
	at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:761)
	at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1105)
	at com.google.api.core.ApiFutures.addCallback(ApiFutures.java:62)
	at bq.DataWriter.append(DataWriter.java:61)
	at bq.DataWriter$AppendCompleteCallback.onFailure(DataWriter.java:109)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:67)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
	at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:94)
	at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:76)
	at com.google.api.core.SettableApiFuture.setException(SettableApiFuture.java:51)
	at com.google.cloud.bigquery.storage.v1.ConnectionWorker.cleanupInflightRequests(ConnectionWorker.java:587)
	at com.google.cloud.bigquery.storage.v1.ConnectionWorker.appendLoop(ConnectionWorker.java:515)
	at com.google.cloud.bigquery.storage.v1.ConnectionWorker.access$000(ConnectionWorker.java:58)
	at com.google.cloud.bigquery.storage.v1.ConnectionWorker$1.run(ConnectionWorker.java:233)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.google.cloud.bigquery.storage.v1.Exceptions$StreamWriterClosedException: FAILED_PRECONDITION: Connection is closed due to com.google.api.gax.rpc.InternalException: io.grpc.StatusRuntimeException: INTERNAL: A retriable error could not be retried due to Extensible Stubs memory limits for streams (see go/xs-retries-memory-limit-reached) (old status: RPC::STREAM_BROKEN: Connection to server broken (OnChannelError))
	at com.google.cloud.bigquery.storage.v1.ConnectionWorker.appendInternal(ConnectionWorker.java:309)
	at com.google.cloud.bigquery.storage.v1.ConnectionWorker.append(ConnectionWorker.java:267)
	at com.google.cloud.bigquery.storage.v1.StreamWriter$SingleConnectionOrConnectionPool.append(StreamWriter.java:130)
	at com.google.cloud.bigquery.storage.v1.StreamWriter.append(StreamWriter.java:354)
	at com.google.cloud.bigquery.storage.v1.JsonStreamWriter.append(JsonStreamWriter.java:232)
	at com.google.cloud.bigquery.storage.v1.JsonStreamWriter.append(JsonStreamWriter.java:113)
	at bq.DataWriter.append(DataWriter.java:60)
	... 15 more

This is my build.gradle dependency for storage write api

implementation platform('com.google.cloud:libraries-bom:26.1.3')
implementation 'com.google.cloud:google-cloud-bigquerystorage'
implementation 'com.google.cloud:google-cloud-bigquery'

I am using spring boot version 2.7.2

For code implementation snippet please have a look at this issue: BQ code Snippet
@product-auto-label product-auto-label bot added the api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. label Nov 14, 2022
@Neenu1995 Neenu1995 added the status: duplicate Duplicate. label Nov 15, 2022
@Neenu1995
Copy link
Contributor

Closing the duplicate issue as the original is still open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. status: duplicate Duplicate.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@SubhamSinghal @Neenu1995