Using Hail with dsub

[buutrg]

Hi all, I am trying to use hail via dsub to extract a subset of variants in All of Us server. I think this is the most relevant image I can use https://github.com/DataBiosphere/terra-docker/tree/master/terra-jupyter-hail

But it result in error that pyspark is not found. I tried to install pyspark from https://dlcdn.apache.org/spark/spark-3.1.3/spark-3.1.3-bin-hadoop3.tgz. Now it says No FileSystem for scheme "gs".

May I ask do you have any idea how to use hail via dsub?
Your help is really appreciated!

[wnojopra]

It sounds like you're running dsub on the AoU platform. From this Aou Support Article, "Within the Researcher Workbench, internet access is restricted from batch VMs. With the exception of Google APIs, VMs are unable to send or receive network traffic including files, APIs, or packages/code". This isn't specifically a dsub issue. Please reach out to AoU support for help installing pyspark in that image.