trchudley · trchudley · Oct 7, 2024 · Oct 7, 2024 · Oct 7, 2024 · Oct 7, 2024
diff --git a/.github/workflows/unit_test.yml b/.github/workflows/unit_test.yml
@@ -14,6 +14,7 @@ jobs:
       matrix:
         python-version:
           - "3.10"
+          - "3.11"
 
     steps:
       - name: Checkout repo

diff --git a/README.md b/README.md
@@ -40,16 +40,15 @@ Please visit the [pDEMtools readthedocs](https://pdemtools.readthedocs.io/en/lat
 Update when v.1.0 uploaded to Zendoo.
  -->
 
- The use of the pDEMtools package can be cited as follows:
+A software paper is being prepared for the [Journal of Open Source Software](https://joss.theoj.org/). In the meantime, the use of the pDEMtools package can be cited as follows:
 
-
-> Chudley, T. R. and Howat, I. M. (2024) pDEMtools (vX.X.X). GitHub. https://github.com/trchudley/pDEMtools
+> Chudley, T. R. and Howat, I. M. (2024) pDEMtools: conveniently search, download, and process ArcticDEM and REMA products (vX.X.X). GitHub. https://github.com/trchudley/pDEMtools
 
 or by using `bibtex`:
 
 ```
 @software{pDEMtools
-   author = {Chudley, Thomas R. and Howat, Ian M.}, title = {pDEMtools}, year = 2024, publisher = {GitHub}, version = {X.X.X}, url = {https://github.com/trchudley/pDEMtools} 
+   author = {Chudley, Thomas R. and Howat, Ian M.}, title = {pDEMtools: conveniently search, download, and process ArcticDEM and REMA products}, year = 2024, publisher = {GitHub}, version = {X.X.X}, url = {https://github.com/trchudley/pDEMtools} 
 }
 ```
 

diff --git a/docs/appendix/faq.md b/docs/appendix/faq.md
@@ -0,0 +1,13 @@
+# FAQ
+
+This page records useful questions and answers raised by users of `pdemtools`, on the [GitHub issues page](https://github.com/trchudley/pdemtools/issues) and elsewhere. If you have any further questions, do not hesitate to raise them via GitHub issue!
+
+> __1. How large an area is appropriate to work with? I was lacking was some guidance regarding how big of an area is ‘appropriate’ to work with using the software. I several times ran into memory issues (running it on a new laptop) because I had selected an area that was too large.__
+
+To an extent, 'how big of an area can I work with' is basically a question of how much RAM you have available, your internet speed, and your patience - `pdemtools` won't try and stop you from downloading a 2 m mosaic of the whole of Antarctica! 
+
+However, the tool - as it currently stands - is designed around the challenge of downloading time-series stacks of specific study sites for change analysis (e.g. batch download and coregistration to the same datum). These sites are generally _smaller_ than the size of strip data (generally, ArcticDEM and REMA strips are of the order of ~10 km wide and ~100 km long). This design philosophy won't necessarily make things more convenient for other tasks. For example, if you wanted to download every REMA strip over the Larsen C ice shelf, (i) the `aoi_frac` variable in `search()` probably won't be very convenient, as all individual strips will be much smaller than your AOI; (ii) once they're downloaded, many strips will have no stable ground to coregister against (although ICESat-2 coregistration is planned as a forthcoming feature); (iii) given the very large AOI, the single contiguous images output by workflows such as the `batch_download_and_coregister.py` script  will be (for mosaics) inordinately large and (for strips) largely `NaN` data. As such, the workflows presented in the `pdemtools` readthedocs are best tuned to the needs of users who are looking at contexts similar to those in the examples: study sites on the order of 10s kms in size. These notebooks, in our experience, seem to run succesfully on a range of laptops/desktops outside an HPC environment.
+
+`pdemtools` will still be useful in some ways for these larger-scale tasks. For instance, the dataframe returned by the `search` tool can providing you a list of download urls for strip tarballs (accessed via the dataframe's `'fileurl'` column) that can then be retrieved via `wget` or similar. Or the `mosaic()` function can return mosaics on a custom tile grid suited to your own requirements if the standard PGC tiles aren't useful for your needs.
+
+We would encourage users who are interested in larger-scale analysis to get in touch with their needs if they find `pdemtools` doesn't quite suit. We have the goal of making `pdemtools` a one-stop-shop for ArcticDEM/REMA DEM analysis, and other perspectives and requirements are an essential part of this.
diff --git a/docs/appendix/version_updates.md b/docs/appendix/version_updates.md
@@ -1,10 +1,11 @@
 # Version updates
 
+From the public conda release of version 0.8.0 onwards, [version updates and notes are automatically recorded on the GitHub repo 'releases' page`](https://github.com/trchudley/pdemtools/releases). 
+
+Versions prior to this are recorded in the table below for posterity:
+
 | Version | Date | Notes |
 | ------- | ---- | ----- |
-| 0.8.2 | June 2024 | Bug fix to address fail loading of mosaics on Windows systems |
-| 0.8.1 | June 2024 | Minor modification to hillshade script to allow for MacGregor _et al._ (2024) flow-aware hillshade. Supplementary notebook available in GitHub `notebooks` directory. |
-| 0.8.0 | May 2024 | Iceberg detection now corrects for sea surface; final additions to documentation before public conda/pip release.
 | 0.7.0 | March 2024 | Minor bug fixes and more detailed docstrings to support readthedocs | 
 | 0.6.0 | February 2024 | Added `data.bedrock_mask_from_vector()` function. |
 | 0.5.0 | January 2024 | Fixed bug in `_coreg.py` to improve Nuth and Kääb (2011)-style coregistration. |

diff --git a/docs/getting_started/cite.md b/docs/getting_started/cite.md
@@ -2,13 +2,13 @@
 
 A software paper is being prepared for the [Journal of Open Source Software](https://joss.theoj.org/). In the meantime, the use of the pDEMtools package can be cited as follows:
 
-> Chudley, T. R. and Howat, I. M. (2024) pDEMtools (vX.X.X). GitHub. https://github.com/trchudley/pDEMtools
+> Chudley, T. R. and Howat, I. M. (2024) pDEMtools: conveniently search, download, and process ArcticDEM and REMA products (vX.X.X). GitHub. https://github.com/trchudley/pDEMtools
 
 or by using `bibtex`:
 
 ```
 @software{pDEMtools
-   author = {Chudley, Thomas R. and Howat, Ian M.}, title = {pDEMtools}, year = 2024, publisher = {GitHub}, version = {X.X.X}, url = {https://github.com/trchudley/pDEMtools} 
+   author = {Chudley, Thomas R. and Howat, Ian M.}, title = {pDEMtools: conveniently search, download, and process ArcticDEM and REMA products}, year = 2024, publisher = {GitHub}, version = {X.X.X}, url = {https://github.com/trchudley/pDEMtools} 
 }
 ```
 

diff --git a/docs/getting_started/install.md b/docs/getting_started/install.md
@@ -12,6 +12,8 @@ or with `mamba`:
 mamba install pdemtools -c conda-forge
 ```
 
+If you don't already have `conda` or `mamba` installed, the recommended method is via `micromamba`. Instructions can be found [here](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html).
+
 It is also possible to install `pdemtools` with `pip`:
 
 ```bash

diff --git a/docs/getting_started/supplementary_datasets.md b/docs/getting_started/supplementary_datasets.md
@@ -1,11 +1,19 @@
 # Supplementary datasets
 
-To make the most of pDEMtools, two supplementary datasets must available locally:
+To make the most of pDEMtools, two supplementary datasets must available locally.
 
 ## ArcticDEM/REMA strip index
 
-The first is the ArcticDEM or REMA strip index made available by the PGC  (in GeoParquet format), used by the `search` function. Strip index GeoParquet files can be downloaded from the PGC ([Greenland](https://data.pgc.umn.edu/elev/dem/setsm/ArcticDEM/indexes/), [Antarctica](https://data.pgc.umn.edu/elev/dem/setsm/REMA/indexes/)). To enable rapid searching, *please download the GeoParquet file format*: these files end in `_gpqt.zip`. Unzip them before use.
+The first dataset is the ArcticDEM or REMA strip index made available by the PGC  (in GeoParquet format), used by the `search` function. Strip index GeoParquet files can be downloaded from the PGC ([Greenland](https://data.pgc.umn.edu/elev/dem/setsm/ArcticDEM/indexes/), [Antarctica](https://data.pgc.umn.edu/elev/dem/setsm/REMA/indexes/)). 
+
+The appriate files to download are the `ArcticDEM_Strip_Index_s2s041_gpqt.zip` and `REMA_Strip_Index_s2s041_gpkg.zip` - note the __GeoParquet file format__ (ending `_gpqt.zip`), *not* the similarly named GeoPackage (ending `_gpkg.zip`). No mosaic files are necessary - these are included with `pdemtools`. 
+
+Unzip the files before before use. They can be placed anywhere in your file system - the filepath, as a string, is provided to the relevant `pdemtools` functions.
+
+> __NOTE:__ These index files are maintained by the PGC, so we cannot guarantee that newly released updates to ArcticDEM/REMA index files will not temporarily break `pdemtools` functions until we can fix and update the software. From version 0.8.3 onwards, these files are known to work up to and including the latest PGC data update (ArcticDEM Strip Release 2023, released Aug 2024).
 
 ## BedMachine
 
-The second is the Greenland BedMachine (v5; Morlighem _et al._ 2022a) or Antarctica BedMachine (v3; Morlighem _et al._ 2022b), which is the default geoid/bedrock mask used by the geoid correction and coregistration functions (NB: for applications outside of the ice sheets, it is possible to use your own geoid/bedrock mask). BedMachine can be downloaded from the NSIDC ([Greenland](https://nsidc.org/data/idbmg4/versions/5), [Antarctica](https://nsidc.org/data/nsidc-0756/versions/3)).
+The second dataset is the Greenland BedMachine (v5; Morlighem _et al._ 2022a) or Antarctica BedMachine (v3; Morlighem _et al._ 2022b) in netcdf format. This dataset provides the default EIGEN-6C4 geoid and ice/bedrock mask used by the geoid correction and coregistration functions (NB: for applications outside of the ice sheets, it is possible to use your own geoid/bedrock mask). 
+
+The appropriate versions of BedMachine can be downloaded from the NSIDC ([Greenland](https://nsidc.org/data/idbmg4/versions/5), [Antarctica](https://nsidc.org/data/nsidc-0756/versions/3)). Follow these links to the NSIDC website. Once you're there, the most convenient way of downloading is via the 'Data Access Tool'. __Ensure you download the netcdf version__ (ending `*.nc`), not the tif version.
diff --git a/docs/index.rst b/docs/index.rst
@@ -19,11 +19,11 @@ pDEMtools
   :align: center
   :alt: A hillshaded DEM of Helheim Glacier
 
-pDEMtool provides a convenient set of functions to explore, download, and preprocess high-resolution DEMs of the polar regions from the ArcticDEM (Porter *et al.*  2022; 2023) and Reference Elevation Model of Antarctica (REMA; Howat *et al.* 2022a, b) products, courtesy of the Polar Geospatial Center (PGC).
+``pdemtools`` provides a convenient set of functions to explore, download, and preprocess high-resolution DEMs of the polar regions from the ArcticDEM (Porter *et al.*  2022; 2023) and Reference Elevation Model of Antarctica (REMA; Howat *et al.* 2022a, b) products, courtesy of the Polar Geospatial Center (PGC).
 
-The first aim of pDEMtools is to enable access to ArcticDEM and REMA mosaics and multitemporal strips using the ``search()`` function and ``load`` module:
+The first aim of ``pdemtools`` is to enable access to ArcticDEM and REMA mosaics and multitemporal strips using the ``search()`` function and ``load`` module:
 
- - ``search()``: This function aims to replicate the kind of convenient catalogue searching available when querying a dynamic STAC catalogue (e.g. ``pystac_client``), allowing users to easily find relevant ArcticDEM and REMA strips for their areas of interest. 
+ - ``search()``: This function aims to provide a convenient way of searching the ArcticDEM/REMA strip index - analogous to querying a dynamic STAC catalogue (e.g. via ``pystac_client``) - allowing users to easily find relevant ArcticDEM and REMA strips for their areas of interest. 
  - ``load``: This module provides simple one-line functions to preview and download strips and mosaics from the relevant AWS bucket to an ``xarray`` Dataset.
 
 
@@ -70,6 +70,7 @@ Rather than introducing custom classes, pDEMtools will always try and return DEM
    :glob:
    :caption: Appendix:
 
+   appendix/faq.md
    appendix/community_guidelines.md
    appendix/version_updates.md
    appendix/references.md

diff --git a/environment.yml b/environment.yml
@@ -3,7 +3,7 @@ channels:
   - conda-forge
   - defaults
 dependencies:
-  - python=>3.10
+  - python>=3.10
   - rioxarray 
   - geopandas 
   - opencv 

diff --git a/images/example_dem_difference.jpg b/images/example_dem_difference.jpg
diff --git a/images/example_iceberg_height.jpg b/images/example_iceberg_height.jpg
diff --git a/images/example_mosaic_terrain.jpg b/images/example_mosaic_terrain.jpg
diff --git a/notebooks/flow_aware_hillshade.ipynb b/notebooks/flow_aware_hillshade.ipynb
diff --git a/notebooks/get_icebergs.ipynb b/notebooks/get_icebergs.ipynb
diff --git a/notebooks/mosaic_and_terrain.ipynb b/notebooks/mosaic_and_terrain.ipynb