20200716 developer call notes

[hammer]

Pre-notes

Process and tools

• Submit an abstract to Genome Informatics? Due 21 July 2020. They need them!
• Quansight
• Docs: sgkit#19

IO

• From Jerome: what are we thinking about on disk storage? I assume we're planning on using zarr via xarray's support: http://xarray.pydata.org/en/stable/io.html#zarr. However, I think there's a bit of work to be done in tuning the chunk sizes, providing a good interface for multithreaded writing, choosing the appropriate backend, etc that it would be good to talk through. I ended up doing a fair bit of work on top of zarr for tsinfer's input data formats, which we might be able to reuse some of: https://github.com/tskit-dev/tsinfer/blob/master/tsinfer/formats.py. I'm hoping to get rid of these formats and use sgkit within tsinfer, so being able to iterate efficiently over very large genotype arrays across both dimensions is a key application for me.
• VCF: where should the vcfzarr function live? sgkit#40
• BGEN: needs agreement on dosage data structure (see below) before merging.

Data structures

• Dosage: how to represent biallelic and multiallelic data? sgkit#21
• Document: sgkit#42

Dispatching

Core operations

• count_alleles: sgkit#36
• linreg: sgkit#16
• API design: next steps on xarray and “wrapper” APIs sgkit#16 sgkit#43

Notes

Attendees

• Jeff Hammerbacher
• Jerome Kelleher
• Alistair Miles
• Eric Czech
• Tom White
• Rafal Wojdyla

Discussion

• Genome Informatics: we’re too early
• Quansight: have them attend calls, and find work for them
  • Alistair out next week
  • Alistair’s stories: PBS selection, PCA
  • Alistair has a JupyterHub w/ Dask + Kubernetes integration
• Docs: sgkit#19
  • Don’t block on Jeff any longer!
  • Be sure to run doctest in our CI
  • Jerome notes there are linters for docstrings we should use
  • Use numpy/numpydoc Sphinx extension to parse docstrings?
  • One wrinkle: don’t duplicate types in docstrings
  • Investigate sphinx-contrib/napoleon Sphinx extension to support both Google style and NumPy style docstrings?
  • Doc search comment
  • Jerome feels RTD has been getting worse, and since we’re building docs in CI, maybe just push to a host of our own
  • GitHub pages to host at first?
  • Alistair notes RTD gives a stable URI for each version
  • Tom will pick up and carry
• Code coverage now in
• Rafal improving things too
• IO: we will use xarray’s zarr support, but there’s still more work to be done
  • Jerome wrote tsinfer/formats.py for his use case
  • What backend do we use with Zarr? Jerome using LMDB
  • Xarray’s use of Zarr has some expectations of grouping and attributes
  • Eric notes that to write, we need to make sure chunk sizes are the same across every dimension in every array, and we will likely want to override default compression choices
    • TODO: find or report an issue
  • Convert to native format should be a well-documented pattern
  • Alistair notes it will be important to parallelize the conversion, but different formats may require different strategies
    • Alistair uses a different group for each contig for VCF
    • Jerome notes you can use indexes to parallelize; can’t parallelize by squares, only rectangles!
    • Could also chunk by genome size regions, but then don’t know how many rows will be in that block
    • Tom notes that he’s worked on it in Spark
• VCF: where should the vcfzarr function live? sgkit#40
  • Alistair: put in main repo
    • Doesn’t deal w/ ordinary VCFs
    • It’s not much code
  • Jerome: put in sgkit-vcf repo
    • VCF files in sgkit main repo are bad!
  • Tom notes there are no dependencies on any libraries
  • Rafal notes CI works differently across repos
  • Eric thinks it will be hard to hide VCF details
  • Alistair notes that isolating htslib is why to break out
• Dosage: how to represent biallelic and multiallelic data? sgkit#21
  • Tom doesn’t know the domain well enough to propose a model
  • Alistair has never worked with array data or GWAS studies
  • Alistair notes that there are two different BGEN formats with two different representations of dosages
  • Note UKB has some multiallelci
  • Eric notes there will be missing information
  • Eric notes that GWAS regressions can use information in dosages, not just hard calls
  • Up to user to add missing information
  • Eric notes Hail’s BGEN interaction is a good one
  • Alistair comes from WGS data and wants to do PCA; needs to turn hard calls into a continuous variable. Must deal with multi-allelic sites and missing calls. How to distinguish homozygous reference calls and missing calls when converting to dosages?
• linreg: sgkit#16
  • Eric just needs to fix mypy issues
  • Wrapper discussion can happen at sgkit#43
• count_alleles: sgkit#36
  • Alistair: if we’re happy with the API, put it in, then benchmark the performance and refactor later.
  • Maybe file an issue to benchmark performance later?
• Wrapper API sgkit#43
  • Next step is to get a prototype
  • Eric: we have a pipeline that’s going from messy to clean, and we’re trying to put structure somewhere in the middle. “Schemafication” should be dynamic.
  • We will go forward with the xarray API and explore a lightweight wrapper class that won’t assume structure exists
  • Alistair: how would this API handle a genome range query?
  • Eric notes stateful functions are an example where a high-level API