how to efficiently extract US-Wave timeseries dataset for specific spatiotemporal boundary

[danuriyanto]

Bug Description
I wanted to extract the timeseries data of significant wave height (swh) in the specific area of the Atlantic region. What is the most efficient method to perform this task? I tried using the rex-WaveX extraction and was somewhat stuck in the loop with the following traceback. However, if I call the index individually, it passes through the loop and feeds data.

Is there any method to achieve the task? I am thinking about giving the "box" spatial boundary condition and specify the temporal boundary condition to extract all the timeseries.

Full Traceback

ERROR:root:got <class 'requests.exceptions.RetryError'> exception: HTTPSConnectionPool(host='developer.nrel.gov', port=443): Max retries exceeded with url: /api/hsds/datasets/d-3e9a0496-75729c43-eaa5-750126-3f3071?domain=%2Fnrel%2FUS_wave%2FAtlantic%2FAtlantic_wave_1979.h5&api_key=rcufYNIUpUJPlUP5n637v0O03DKEljn5IfyabLFZ (Caused by ResponseError('too many 503 error responses'))
---------------------------------------------------------------------------
MaxRetryError                             Traceback (most recent call last)
File [c:\ProgramData\miniconda3\lib\site-packages\requests\adapters.py:489](file:///C:/ProgramData/miniconda3/lib/site-packages/requests/adapters.py:489), in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    488 if not chunked:
--> 489     resp = conn.urlopen(
    490         method=request.method,
    491         url=url,
    492         body=request.body,
    493         headers=request.headers,
    494         redirect=False,
    495         assert_same_host=False,
    496         preload_content=False,
    497         decode_content=False,
    498         retries=self.max_retries,
    499         timeout=timeout,
    500     )
    502 # Send the request.
    503 else:

File [c:\ProgramData\miniconda3\lib\site-packages\urllib3\connectionpool.py:878](file:///C:/ProgramData/miniconda3/lib/site-packages/urllib3/connectionpool.py:878), in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    877     log.debug("Retry: %s", url)
--> 878     return self.urlopen(
    879         method,
    880         url,
    881         body,
    882         headers,
    883         retries=retries,
    884         redirect=redirect,
    885         assert_same_host=assert_same_host,
    886         timeout=timeout,
    887         pool_timeout=pool_timeout,
    888         release_conn=release_conn,
    889         chunked=chunked,
    890         body_pos=body_pos,
    891         **response_kw
    892     )
    894 return response

File [c:\ProgramData\miniconda3\lib\site-packages\urllib3\connectionpool.py:878](file:///C:/ProgramData/miniconda3/lib/site-packages/urllib3/connectionpool.py:878), in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    877     log.debug("Retry: %s", url)
--> 878     return self.urlopen(
    879         method,
    880         url,
    881         body,
    882         headers,
    883         retries=retries,
    884         redirect=redirect,
    885         assert_same_host=assert_same_host,
    886         timeout=timeout,
    887         pool_timeout=pool_timeout,
    888         release_conn=release_conn,
    889         chunked=chunked,
    890         body_pos=body_pos,
    891         **response_kw
    892     )
    894 return response

    [... skipping similar frames: HTTPConnectionPool.urlopen at line 878 (7 times)]

File [c:\ProgramData\miniconda3\lib\site-packages\urllib3\connectionpool.py:878](file:///C:/ProgramData/miniconda3/lib/site-packages/urllib3/connectionpool.py:878), in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    877     log.debug("Retry: %s", url)
--> 878     return self.urlopen(
    879         method,
    880         url,
    881         body,
    882         headers,
    883         retries=retries,
    884         redirect=redirect,
    885         assert_same_host=assert_same_host,
    886         timeout=timeout,
    887         pool_timeout=pool_timeout,
    888         release_conn=release_conn,
    889         chunked=chunked,
    890         body_pos=body_pos,
    891         **response_kw
    892     )
    894 return response

File [c:\ProgramData\miniconda3\lib\site-packages\urllib3\connectionpool.py:868](file:///C:/ProgramData/miniconda3/lib/site-packages/urllib3/connectionpool.py:868), in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    867 try:
--> 868     retries = retries.increment(method, url, response=response, _pool=self)
    869 except MaxRetryError:

File [c:\ProgramData\miniconda3\lib\site-packages\urllib3\util\retry.py:592](file:///C:/ProgramData/miniconda3/lib/site-packages/urllib3/util/retry.py:592), in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
    591 if new_retry.is_exhausted():
--> 592     raise MaxRetryError(_pool, url, error or ResponseError(cause))
    594 log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)

MaxRetryError: HTTPSConnectionPool(host='developer.nrel.gov', port=443): Max retries exceeded with url: /api/hsds/datasets/d-3e9a0496-75729c43-eaa5-750126-3f3071?domain=%2Fnrel%2FUS_wave%2FAtlantic%2FAtlantic_wave_1979.h5&api_key=rcufYNIUpUJPlUP5n637v0O03DKEljn5IfyabLFZ (Caused by ResponseError('too many 503 error responses'))

During handling of the above exception, another exception occurred:

RetryError                                Traceback (most recent call last)
File [c:\ProgramData\miniconda3\lib\site-packages\h5pyd\_hl\httpconn.py:438](file:///C:/ProgramData/miniconda3/lib/site-packages/h5pyd/_hl/httpconn.py:438), in HttpConn.GET(self, req, format, params, headers, use_cache)
    437 s = self.session
--> 438 rsp = s.get(
    439     self._endpoint + req,
    440     params=params,
    441     headers=headers,
    442     stream=True,
    443     timeout=self._timeout,
    444     verify=self.verifyCert(),
    445 )
    446 self.log.info("status: {}".format(rsp.status_code))

File [c:\ProgramData\miniconda3\lib\site-packages\requests\sessions.py:600](file:///C:/ProgramData/miniconda3/lib/site-packages/requests/sessions.py:600), in Session.get(self, url, **kwargs)
    599 kwargs.setdefault("allow_redirects", True)
--> 600 return self.request("GET", url, **kwargs)

File [c:\ProgramData\miniconda3\lib\site-packages\requests\sessions.py:587](file:///C:/ProgramData/miniconda3/lib/site-packages/requests/sessions.py:587), in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    586 send_kwargs.update(settings)
--> 587 resp = self.send(prep, **send_kwargs)
    589 return resp

File [c:\ProgramData\miniconda3\lib\site-packages\requests\sessions.py:701](file:///C:/ProgramData/miniconda3/lib/site-packages/requests/sessions.py:701), in Session.send(self, request, **kwargs)
    700 # Send the request
--> 701 r = adapter.send(request, **kwargs)
    703 # Total elapsed time of the request (approximately)

File [c:\ProgramData\miniconda3\lib\site-packages\requests\adapters.py:556](file:///C:/ProgramData/miniconda3/lib/site-packages/requests/adapters.py:556), in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    555 if isinstance(e.reason, ResponseError):
--> 556     raise RetryError(e, request=request)
    558 if isinstance(e.reason, _ProxyError):

RetryError: HTTPSConnectionPool(host='developer.nrel.gov', port=443): Max retries exceeded with url: /api/hsds/datasets/d-3e9a0496-75729c43-eaa5-750126-3f3071?domain=%2Fnrel%2FUS_wave%2FAtlantic%2FAtlantic_wave_1979.h5&api_key=rcufYNIUpUJPlUP5n637v0O03DKEljn5IfyabLFZ (Caused by ResponseError('too many 503 error responses'))

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
Cell In[3], line 5
      3 # for i in len(coord_tuples)
      4 for i in range(len(coord_MA)):
----> 5     with WaveX(wave_file, hsds=True) as f:
      6         lat_lon_swh = f.get_lat_lon_df('significant_wave_height', coord_MA[i])
      7     MA_swh[i] = lat_lon_swh 

Cell In[3], line 6
      4 for i in range(len(coord_MA)):
      5     with WaveX(wave_file, hsds=True) as f:
----> 6         lat_lon_swh = f.get_lat_lon_df('significant_wave_height', coord_MA[i])
      7     MA_swh[i] = lat_lon_swh 

File [c:\ProgramData\miniconda3\lib\site-packages\rex\resource_extraction\resource_extraction.py:788](file:///C:/ProgramData/miniconda3/lib/site-packages/rex/resource_extraction/resource_extraction.py:788), in ResourceX.get_lat_lon_df(self, ds_name, lat_lon, check_lat_lon)
    765 def get_lat_lon_df(self, ds_name, lat_lon, check_lat_lon=True):
    766     """
    767     Extract timeseries of site(s) nearest to given lat_lon(s) and return
    768     as a DataFrame
   (...)
    786         Time-series DataFrame for given site(s) and dataset
    787     """
--> 788     gid = self.lat_lon_gid(lat_lon, check_lat_lon=check_lat_lon)
    789     df = self.get_gid_df(ds_name, gid)
    791     return df

File [c:\ProgramData\miniconda3\lib\site-packages\rex\resource_extraction\resource_extraction.py:608](file:///C:/ProgramData/miniconda3/lib/site-packages/rex/resource_extraction/resource_extraction.py:608), in ResourceX.lat_lon_gid(self, lat_lon, check_lat_lon)
    605 dist, gids = self.tree.query(lat_lon)
    607 if check_lat_lon:
--> 608     self._check_lat_lon(lat_lon)
    609     dist_check = dist > self.distance_threshold
    610     if np.any(dist_check):

File [c:\ProgramData\miniconda3\lib\site-packages\rex\resource_extraction\resource_extraction.py:561](file:///C:/ProgramData/miniconda3/lib/site-packages/rex/resource_extraction/resource_extraction.py:561), in ResourceX._check_lat_lon(self, lat_lon)
    552 def _check_lat_lon(self, lat_lon):
    553     """
    554     Check lat lon coordinates against domain
    555 
   (...)
    559         Either a single (lat, lon) pair or series of (lat, lon) pairs
    560     """
--> 561     lat_min, lat_max = np.sort(self.lat_lon[:, 0])[[0, -1]]
    562     lon_min, lon_max = np.sort(self.lat_lon[:, 1])[[0, -1]]
    564     lat = lat_lon[:, 0]

Code Sample

# import
from rex import WaveX
import h5pyd
import pandas as pd
import numpy as np

# reading the coordinate tuple 
coord_MA = pd.read_csv('MA_grid_2km.csv')
coord_MA = [tuple(row) for row in coord_MA.to_records(index=False)]

# list of NREL Hindcast data by year
year = np.arange(1979,2011,1,dtype=int)
wave_file = []
for i in year:
        a =  "/nrel/US_wave/Atlantic/Atlantic_wave_"+str(i)+".h5"
        wave_file.append(a)

MA_swh = pd.DataFrame()

for i in range(len(wave_file)):
    for j in range(len(coord_MA)):
        with WaveX(wave_file[i], hsds=True) as f:
            lat_lon_swh = f.get_lat_lon_df('significant_wave_height', coord_MA[j])
        MA_swh[i] = lat_lon_swh 

To Reproduce
Steps to reproduce the problem behavior

1. Run the code above

Expected behavior
Expected to have the timeseries of 547 locations from 01/01/1979 to 12/31/2010.

Screenshots
When i=3, error with traceback mentioned above pops up. However if I call this code:

with WaveX(wave_file[1], hsds=True) as f:
    lat_lon_swh = f.get_lat_lon_df('significant_wave_height', coord_MA[3])`

The system returns:

System (please complete the following information):

• OS: Windows

Additional context
Here's the grid file called in the script.

I hope someone can help!
Thank you,
Danu

MA_grid_2km.csv