Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the dependency of netCDF4 #3643

Draft
wants to merge 136 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
136 commits
Select commit Hold shift + click to select a range
bcf43f0
Wrap GMT's standard data type GMT_IMAGE for images
seisman Mar 18, 2024
a052a1a
Initial implementation of to_dataarray method for _GMT_IMAGE class
weiji14 Mar 20, 2024
59d523c
pygmt.grdcut: Support both grid and image output
seisman Apr 16, 2024
56a6d65
Merge branch 'main' into datatypes/gmtimage
seisman Apr 17, 2024
3315324
Merge branch 'main' into gmtimage
seisman Apr 19, 2024
cea3374
Fix
seisman Apr 19, 2024
80d9837
Refactor
seisman Apr 19, 2024
22fba56
fix
seisman Apr 19, 2024
f71e79c
Merge branch 'main' into datatypes/gmtimage
weiji14 Jun 18, 2024
4cce4a2
Small typo fixes and add output type-hint for to_dataarray
weiji14 Jun 18, 2024
e02b650
Fix mypy error using np.array([0, 1, 2]) instead of np.arange
weiji14 Jun 18, 2024
f3d4b1f
Parse name and data_attrs from grid/image header
weiji14 Jun 18, 2024
4390136
Transpose array to (band, y, x) order and add doctest for to_dataarray
weiji14 Jun 20, 2024
5f25669
Set registration and gtype from header
weiji14 Jun 20, 2024
a3c6c14
Print basic shape and padding info in _GMT_IMAGE doctest
weiji14 Jun 20, 2024
5888e10
Only set Conventions = CF-1.7 attribute for NetCDF grid type
weiji14 Jun 20, 2024
798e658
Merge branch 'main' into datatypes/gmtimage
weiji14 Jun 20, 2024
3dbf2f2
Remove rioxarray import
weiji14 Jun 20, 2024
3a24ebd
Apply suggestions from code review
seisman Jun 20, 2024
4eee7e6
Merge branch 'main' into gmtimage
seisman Jun 20, 2024
5e390d4
Address reviewer's comments
seisman Jun 20, 2024
003383d
Fix GMT_OUT
seisman Jun 21, 2024
606ac7e
Merge branch 'main' into gmtimage
seisman Jun 21, 2024
c6cdcc8
Merge branch 'main' into gmtimage
seisman Jul 7, 2024
377941a
Revert changes for _GMT_IMAGE
seisman Jul 7, 2024
20617f5
Use rioxarray.open_rasterio for loading images
seisman Jul 7, 2024
a998718
Check if rioxarray is installed
seisman Jul 7, 2024
86cab44
Improve grdcut
seisman Jul 7, 2024
6031bab
Fix typos in grdcut
seisman Jul 7, 2024
eb0af2d
Add tests for grdcut images
seisman Jul 7, 2024
7f6ca7d
Fix one failing test
seisman Jul 7, 2024
21b194a
Fix open_rasterio
seisman Jul 7, 2024
e7eaf5c
Fix open_rasterio
seisman Jul 7, 2024
e3c8569
Make sure the image is loaded
seisman Jul 7, 2024
1c8312c
Update pygmt/clib/session.py
seisman Jul 7, 2024
3913430
Use rioxarray.open_rasterio in a context manager
seisman Jul 8, 2024
812a225
Merge branch 'main' into gmtimage
seisman Jul 8, 2024
90bd29e
Merge remote-tracking branch 'origin/gmtimage' into gmtimage
seisman Jul 8, 2024
ab77187
Fix mypy errors
seisman Jul 8, 2024
6f3e474
Move grdcut image tests to a separate test file
seisman Jul 8, 2024
5b07dd9
Fix copy & paste errors
seisman Jul 8, 2024
31272ab
Run codspeed benchmark for test_grdcut_image_dataarray
seisman Jul 8, 2024
6b860bf
Merge branch 'main' into datatypes/gmtimage
seisman Jul 27, 2024
5a09329
Merge branch 'main' into gmtimage
seisman Aug 5, 2024
279595b
Add the raster_kind function to determine the raster kind
seisman Aug 5, 2024
7def4b5
Simplify the grdcut function
seisman Aug 5, 2024
be175d8
Merge branch 'main' into gmtimage
seisman Sep 19, 2024
0bf9368
Merge branch 'main' into datatypes/gmtimage
seisman Sep 19, 2024
7d437be
Use enum for grid ids
seisman Sep 19, 2024
268e34e
Fix the band. Starting from 1
seisman Sep 19, 2024
86765e1
Refactor the tests for images
seisman Sep 19, 2024
86f3ffa
In np.reshape, a is a position-only parameter
seisman Sep 20, 2024
cc28247
Improve tests
seisman Sep 20, 2024
1e2c973
Fix one failing doctest due to xarray changes
seisman Sep 20, 2024
734dc28
The np.reshape's newshape parameter is deprecated
seisman Sep 20, 2024
919dc00
Define grid IDs using IntEnum instead of Enum
seisman Sep 20, 2024
b1eacf1
Pass the new shape as a positional parameter
seisman Sep 20, 2024
aa4fdc9
Fix failing tests
seisman Sep 20, 2024
c87a3ec
One more fix
seisman Sep 20, 2024
a20d8a2
One more fix
seisman Sep 20, 2024
926427b
Simplify a doctest
seisman Sep 20, 2024
c73328e
Improve the tests
seisman Sep 20, 2024
2825eae
Merge branch 'datatypes/gmtimage' into gmtimage
seisman Sep 20, 2024
bf9275c
Remove the workaround for images
seisman Sep 20, 2024
fb97daa
Convert ctypes array to numpy array using np.ctypeslib.as_array
seisman Sep 20, 2024
15b8d53
Fix the incorrect value due to floating number conversion in sphinter…
seisman Sep 20, 2024
8433e78
Merge branch 'ctypesarray' into datatypes/gmtimage
seisman Sep 20, 2024
3e3a6f3
Update the to_dataarray method to match the codes in GMT_GRID
seisman Sep 20, 2024
12ef40a
image data should has uint8 dtype
seisman Sep 20, 2024
f64fbb8
Further improve the tests
seisman Sep 21, 2024
e9cb0a5
Merge branch 'datatypes/gmtimage' into gmtimage
seisman Sep 21, 2024
4f2ae48
Merge branch 'main' into datatypes/gmtimage
seisman Sep 24, 2024
d49afed
Add a note that currently only 3-band images are supported
seisman Sep 24, 2024
a97d0b3
Apply suggestions from code review
seisman Sep 28, 2024
f70bec0
Merge branch 'main' into datatypes/gmtimage
seisman Sep 28, 2024
2fd13fb
Remove the old GMTGridID enums from pygmt/datatypes/header.py
seisman Sep 28, 2024
9972ba1
A minor fix
seisman Sep 28, 2024
ac6b7c3
Merge branch 'datatypes/gmtimage' into gmtimage
seisman Sep 28, 2024
7c32d41
Merge branch 'main' into gmtimage
seisman Sep 29, 2024
9ec00be
Let _raster_kind return grid by default
seisman Sep 29, 2024
f3a2f8e
Simplify the grdcut image tests
seisman Sep 29, 2024
3c12e2b
Add one more test for file in & file out
seisman Sep 29, 2024
f852b0d
Fix typos
seisman Sep 29, 2024
5f7683c
Merge branch 'main' into gmtimage
seisman Sep 30, 2024
bb1a0b0
Use the new load_blue_marble function
seisman Sep 30, 2024
584b5af
Drop the spatial_ref coord
seisman Sep 30, 2024
d75b44f
Remove netCDF4 to see how if affects our tests
seisman Nov 22, 2024
c29d653
Refactor load_static_earth_relief to use GMT for reading, not xarray
seisman Nov 22, 2024
b19ba00
Merge branch 'main' into gmtimage
seisman Nov 22, 2024
cceb929
Merge branch 'main' into gmtimage
seisman Nov 25, 2024
fac51f1
Update _raster_kind
seisman Nov 25, 2024
d913c86
Add pygmt.read to read a dataset/grid/image into pandas.DataFrame/xar…
seisman Dec 2, 2024
f456bf8
Set GMT accessor
seisman Dec 5, 2024
c3cbb6e
Need to set 'source' encoding to make GMT accessor work
seisman Dec 5, 2024
f2a4ce4
Merge branch 'main' into feature/read
seisman Dec 5, 2024
1dd97c6
Fix the source encoding
seisman Dec 5, 2024
7790ea3
No need to set the source encoding in load_remote_dataset.py
seisman Dec 5, 2024
e588008
Revert changes in pygmt/datasets/load_remote_dataset.py
seisman Dec 6, 2024
40d12ee
Improve docstring in pygmt/helpers/testing.py
seisman Dec 6, 2024
fa1021d
Improve docstrinbgs
seisman Dec 6, 2024
c378225
Get rid of decorators
seisman Dec 8, 2024
7b749e0
Improve comment
seisman Dec 8, 2024
8befa58
Get rid of the fmt_docstring alias
seisman Dec 8, 2024
a758752
Fix type hints issue with overload
seisman Dec 9, 2024
9d66cf4
Remove the type ignore flag
seisman Dec 9, 2024
a05383a
region defaults to None
seisman Dec 9, 2024
6ca4ef2
Merge branch 'main' into feature/read
seisman Dec 9, 2024
7851ced
Improve type hints and add tests
seisman Dec 9, 2024
084b87a
Improve the checking of return value of which
seisman Dec 9, 2024
b21997c
Use the read funciton in pygmt/tests/test_datatypes_dataset.py
seisman Dec 9, 2024
5df527d
Merge branch 'feature/read' into dep/netcdf
seisman Dec 9, 2024
c481c14
Fix merge conflict
seisman Dec 9, 2024
a812317
Use the read function instead of the load_dataarray method
seisman Dec 9, 2024
1f0f158
Add one test to make sure that read and load_dataarray returns the sa…
seisman Dec 9, 2024
957c7eb
Simplify pygmt/tests/test_clib_read_data.py with read
seisman Dec 9, 2024
33c3b6e
Merge branch 'feature/read' into dep/netcdf
seisman Dec 9, 2024
6aef3ca
Fix a typo
seisman Dec 9, 2024
517ee52
Revert "Update _raster_kind"
seisman Dec 9, 2024
d5ec308
Merge branch 'main' into gmtimage
seisman Dec 9, 2024
7ca58fb
Remove the _raster_kind function
seisman Dec 9, 2024
66580d4
Add the 'kind' parameter
seisman Dec 9, 2024
194ee90
Minor update
seisman Dec 9, 2024
d5a9732
Merge branch 'feature/read' into dep/netcdf
seisman Dec 9, 2024
ccd9828
Merge branch 'gmtimage' into dep/netcdf
seisman Dec 9, 2024
2f37264
netCDF now is a package used in testing only
seisman Dec 9, 2024
72afbfe
Replace xr.open_dataarray with read
seisman Dec 9, 2024
473428c
Avoid keep the file open
seisman Dec 9, 2024
03de9b7
Fix a typo
seisman Dec 9, 2024
84b2ed4
Remove one unnecessary pytest.skipif marker
seisman Dec 9, 2024
7968096
Add it back because we still need rioxarray
seisman Dec 9, 2024
a300fb9
Merge branch 'gmtimage' into dep/netcdf
seisman Dec 9, 2024
d3167e9
Merge branch 'feature/read' into dep/netcdf
seisman Dec 9, 2024
2e14ff3
Merge branch 'main' into dep/netcdf
seisman Dec 19, 2024
e33b537
Refactor test_accessor to make netCDF4 optional
seisman Dec 19, 2024
f8a7a98
Merge branch 'main' into dep/netcdf
seisman Feb 20, 2025
337dcb3
Fix stylling
seisman Feb 20, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/benchmarks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,6 @@ jobs:
numpy
pandas
xarray
netCDF4
packaging
geopandas
pyarrow-core
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/cache_data.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,6 @@ jobs:
numpy
pandas
xarray
netCDF4
packaging
python-build

Expand Down
1 change: 0 additions & 1 deletion .github/workflows/ci_docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,6 @@ jobs:
numpy
pandas
xarray
netCDF4
packaging
contextily
geopandas
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/ci_doctests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,6 @@ jobs:
numpy
pandas
xarray
netCDF4
packaging
contextily
geopandas
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/ci_tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,6 @@ jobs:
numpy=${{ matrix.numpy-version }}
pandas${{ matrix.pandas-version }}
xarray${{ matrix.xarray-version }}
netCDF4
packaging
make
pip
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/ci_tests_dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -146,8 +146,8 @@ jobs:
run: |
python -m pip install --pre --prefer-binary \
--extra-index https://pypi.anaconda.org/scientific-python-nightly-wheels/simple \
numpy pandas xarray netCDF4 packaging \
build contextily dvc geopandas ipython pyarrow rioxarray \
numpy pandas xarray packaging \
build contextily dvc geopandas ipython pyarrow rioxarray netCDF4 \
pytest pytest-cov pytest-doctestplus pytest-mpl pytest-rerunfailures \
sphinx-gallery

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci_tests_legacy.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -61,13 +61,13 @@ jobs:
numpy<2
pandas
xarray
netCDF4
packaging
contextily
geopandas
ipython
pyarrow-core
rioxarray
netCDF4
sphinx-gallery
make
pip
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/type_checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ jobs:
# 3. type checker and stub packages
# 4. other packages that are used somewhere in PyGMT
python -m pip install \
numpy pandas xarray netcdf4 packaging \
numpy pandas xarray packaging \
contextily geopandas ipython pyarrow rioxarray \
mypy pandas-stubs pyarrow-stubs \
matplotlib pytest
Expand Down
1 change: 0 additions & 1 deletion ci/requirements/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ dependencies:
- numpy
- pandas
- xarray
- netCDF4
- packaging
# Optional dependencies
- contextily
Expand Down
1 change: 1 addition & 0 deletions doc/api/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,7 @@ Input/output
:toctree: generated

load_dataarray
read

GMT Defaults
------------
Expand Down
5 changes: 2 additions & 3 deletions doc/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,6 @@ PyGMT requires the following packages to be installed:
- [NumPy](https://numpy.org)
- [pandas](https://pandas.pydata.org)
- [Xarray](https://xarray.dev/)
- [netCDF4](https://unidata.github.io/netcdf4-python)
- [packaging](https://packaging.pypa.io)

:::{note}
Expand Down Expand Up @@ -128,14 +127,14 @@ installed (we'll call it `pygmt` but feel free to change it to whatever you want
::: {tab-item} mamba
:sync: mamba
```
mamba create --name pygmt python=3.13 numpy pandas xarray netcdf4 packaging gmt
mamba create --name pygmt python=3.13 numpy pandas xarray packaging gmt
```
:::

::: {tab-item} conda
:sync: conda
```
conda create --name pygmt python=3.13 numpy pandas xarray netcdf4 packaging gmt
conda create --name pygmt python=3.13 numpy pandas xarray packaging gmt
```
:::
::::
Expand Down
3 changes: 2 additions & 1 deletion environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ dependencies:
- numpy>=1.25
- pandas>=2.0
- xarray>=2023.04
- netCDF4
- packaging
# Optional dependencies
- contextily
Expand All @@ -34,6 +33,8 @@ dependencies:
- pytest-cov
- pytest-doctestplus
- pytest-mpl
# Dev dependencies (testing)
- netCDF4
# Dev dependencies (building documentation)
- geodatasets
- myst-nb
Expand Down
1 change: 1 addition & 0 deletions pygmt/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@
makecpt,
nearneighbor,
project,
read,
select,
sph2grd,
sphdistance,
Expand Down
6 changes: 2 additions & 4 deletions pygmt/datasets/samples.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@
import pandas as pd
import xarray as xr
from pygmt.exceptions import GMTInvalidInput
from pygmt.io import load_dataarray
from pygmt.src import which
from pygmt.src import read, which


def _load_japan_quakes() -> pd.DataFrame:
Expand Down Expand Up @@ -203,8 +202,7 @@ def _load_earth_relief_holes() -> xr.DataArray:
The Earth relief grid. Coordinates are latitude and longitude in degrees. Relief
is in meters.
"""
fname = which("@earth_relief_20m_holes.grd", download="c")
return load_dataarray(fname, engine="netcdf4")
return read("@earth_relief_20m_holes.grd", kind="grid") # type: ignore[return-value]


class GMTSampleData(NamedTuple):
Expand Down
13 changes: 6 additions & 7 deletions pygmt/helpers/testing.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@
import string
from pathlib import Path

import xarray as xr
from pygmt.exceptions import GMTImageComparisonFailure
from pygmt.io import load_dataarray
from pygmt.src import which
from pygmt.src import read


def check_figures_equal(*, extensions=("png",), tol=0.0, result_dir="result_images"):
Expand Down Expand Up @@ -144,17 +144,16 @@ def wrapper(*args, ext="png", request=None, **kwargs):
return decorator


def load_static_earth_relief():
def load_static_earth_relief() -> xr.DataArray:
"""
Load the static_earth_relief file for internal testing.
Load the static_earth_relief.nc file for internal testing.

Returns
-------
data : xarray.DataArray
data
A grid of Earth relief for internal tests.
"""
fname = which("@static_earth_relief.nc", download="c")
return load_dataarray(fname)
return read("@static_earth_relief.nc", kind="grid") # type: ignore[return-value]


def skip_if_no(package):
Expand Down
1 change: 1 addition & 0 deletions pygmt/src/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
from pygmt.src.plot3d import plot3d
from pygmt.src.project import project
from pygmt.src.psconvert import psconvert
from pygmt.src.read import read
from pygmt.src.rose import rose
from pygmt.src.select import select
from pygmt.src.shift_origin import shift_origin
Expand Down
45 changes: 31 additions & 14 deletions pygmt/src/grdcut.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,24 @@
grdcut - Extract subregion from a grid.
"""

from typing import Literal

import xarray as xr
from pygmt.clib import Session
from pygmt.exceptions import GMTInvalidInput
from pygmt.helpers import (
GMTTempFile,
build_arg_list,
data_kind,
fmt_docstring,
kwargs_to_strings,
use_alias,
)
from pygmt.io import load_dataarray

__doctest_skip__ = ["grdcut"]


@fmt_docstring
@use_alias(
G="outgrid",
R="region",
J="projection",
N="extend",
Expand All @@ -28,9 +29,11 @@
f="coltypes",
)
@kwargs_to_strings(R="sequence")
def grdcut(grid, **kwargs) -> xr.DataArray | None:
def grdcut(
grid, kind: Literal["grid", "image"] = "grid", outgrid: str | None = None, **kwargs
) -> xr.DataArray | None:
r"""
Extract subregion from a grid.
Extract subregion from a grid or image.

Produce a new ``outgrid`` file which is a subregion of ``grid``. The
subregion is specified with ``region``; the specified range must not exceed
Expand All @@ -48,6 +51,10 @@
Parameters
----------
{grid}
kind
The raster data kind. Valid values are ``grid`` and ``image``. When the input
``grid`` is a file name, it's hard to determine if the file is a grid or an
image, so we need to specify the kind explicitly. The default is ``grid``.
{outgrid}
{projection}
{region}
Expand Down Expand Up @@ -100,13 +107,23 @@
>>> # 12° E to 15° E and a latitude range of 21° N to 24° N
>>> new_grid = pygmt.grdcut(grid=grid, region=[12, 15, 21, 24])
"""
with GMTTempFile(suffix=".nc") as tmpfile:
with Session() as lib:
with lib.virtualfile_in(check_kind="raster", data=grid) as vingrd:
if (outgrid := kwargs.get("G")) is None:
kwargs["G"] = outgrid = tmpfile.name # output to tmpfile
lib.call_module(
module="grdcut", args=build_arg_list(kwargs, infile=vingrd)
)
# Determine the output data kind based on the input data kind.
match inkind := data_kind(grid):
case "image" | "grid":
outkind = inkind
case "file":
outkind = kind
case "_":
msg = f"Unsupported data type {type(grid)}."
raise GMTInvalidInput(msg)

Check warning on line 118 in pygmt/src/grdcut.py

View check run for this annotation

Codecov / codecov/patch

pygmt/src/grdcut.py#L117-L118

Added lines #L117 - L118 were not covered by tests

return load_dataarray(outgrid) if outgrid == tmpfile.name else None
with Session() as lib:
with (
lib.virtualfile_in(check_kind="raster", data=grid) as vingrd,
lib.virtualfile_out(kind=outkind, fname=outgrid) as voutgrd,
):
kwargs["G"] = voutgrd
lib.call_module(module="grdcut", args=build_arg_list(kwargs, infile=vingrd))
return lib.virtualfile_to_raster(
vfname=voutgrd, kind=outkind, outgrid=outgrid
)
Loading
Loading