You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The following post was exported from discuss.hail.is, a forum for asking questions about Hail which has since been deprecated.
(Apr 19, 2023 at 11:30) megamad said:
Hi forum,
We have a very specific and annoying error, so I have to start with the description of our setup.
We use Hail on a Spark cluster living on top of OpenStack VMs. Besides them we have a HPC cluster that works with Lustre file system where all our active data sit. Since we don’t want to copy data between HPC and OpenStack we have a neat solution to access Lustre from VMs that is called Secure Lustre. We use provided for each project boot image and network to create VMs. Thanks to Secure Lustre we have a specific share of Lustre mouted to /lustre in each VM created in a described way. As a result we don’t use HDFS at all.
Ok, to use lustre within Hail we need to prepend all paths with file:// as in
Please note that the path to json in the error does not have file:// prefix. Indeed file /lustre/path/to/data/dataset.mt/index/part-00000-0f98f82e-3293-40fd-8fb4-d41990c8ac58.idx/metadata.json.gz
does not exist (because this is a file on HDFS), but this file does exist file:///lustre/path/to/data/dataset.mt/index/part-00000-0f98f82e-3293-40fd-8fb4-d41990c8ac58.idx/metadata.json.gz
Thank you so much for this fix.
I’ve just checked it on a fresh Hail 0.2.127 cluster, and everything works like a charm. I no longer get File does not exist error when doing previously described filters.
The text was updated successfully, but these errors were encountered:
Note
The following post was exported from discuss.hail.is, a forum for asking questions about Hail which has since been deprecated.
(Apr 19, 2023 at 11:30) megamad said:
Hi forum,
We have a very specific and annoying error, so I have to start with the description of our setup.
We use Hail on a Spark cluster living on top of OpenStack VMs. Besides them we have a HPC cluster that works with Lustre file system where all our active data sit. Since we don’t want to copy data between HPC and OpenStack we have a neat solution to access Lustre from VMs that is called Secure Lustre. We use provided for each project boot image and network to create VMs. Thanks to Secure Lustre we have a specific share of Lustre mouted to
/lustre
in each VM created in a described way. As a result we don’t use HDFS at all.Ok, to use lustre within Hail we need to prepend all paths with
file://
as inSo in our case every node tries to access their local storage but gets magically redirected to shared Lustre.
Everything works fine unless we try to filter rows by locus. For some reason this won’t work:
They produce a very long nasty error. Here is a part of it that I think makes sense:
Please note that the path to json in the error does not have
file://
prefix. Indeed file/lustre/path/to/data/dataset.mt/index/part-00000-0f98f82e-3293-40fd-8fb4-d41990c8ac58.idx/metadata.json.gz
does not exist (because this is a file on HDFS), but this file does exist
file:///lustre/path/to/data/dataset.mt/index/part-00000-0f98f82e-3293-40fd-8fb4-d41990c8ac58.idx/metadata.json.gz
Interestingly, these will work
As you see filtering by locus is a rain dance for us because we never know what will work and what will lead to
File does not exist
error.I hope and suspect there is a bug in a
MatrixTable.filter_rows
function that omitsfile://
prefix in some cases.We would be very gratefull if someone could find that bug or clarify the situation.
(Apr 19, 2023 at 13:20) tpoterba said:
Is there additional java stack trace you can share? I want to see where in the Hail backend this is coming from.
(Apr 20, 2023 at 09:16) megamad said:
tpoterba thank you for looking into this!
It is an insanely long error, and since I couldn’t find a way to attach a file, I post a link to Google Drive where you can find a full error message:
https://drive.google.com/file/d/16vO88LbnVaP88soiXaTGhxSrqtBgF29s/viewHere are just first 411 lines of the error message below:
(Nov 09, 2023 at 18:05) danking said:
Hey megamad ,
Apologies that this got dropped. I’ve created a GitHub issue to track the bug [query] hail does not preserve schemes on URLs when using the HadoopFS in the SparkBackend · Issue #13998 · hail-is/hail · GitHub . I suspect there’s a simple fix.
(Dec 18, 2023 at 19:03) danking said:
A fix has been merged, once 0.2.127 is released, I’ll let you know.
(Jan 12, 2024 at 18:29) danking said:
This is hopefully fixed and released in 0.2.127. Hail | Change Log And Version Policy
(Mar 11, 2024 at 15:28) megamad said:
Hi danking !
Thank you so much for this fix.
I’ve just checked it on a fresh Hail 0.2.127 cluster, and everything works like a charm. I no longer get
File does not exist
error when doing previously described filters.The text was updated successfully, but these errors were encountered: