Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added fast processing for RINEX 3 #85

Open
wants to merge 27 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
57b74e0
added fast processing to rinex v3
breid-phys May 19, 2022
a429c12
Update test_rinex.py
breid-phys May 19, 2022
1e025c1
moved file to correct place
breid-phys May 26, 2022
9c65502
Delete obs3.py
breid-phys May 26, 2022
f521b9e
Updated rio.py .Z
breid-phys May 26, 2022
254d8e1
fixed critical indexing mistake in fast obs3
breid-phys Jul 7, 2022
8503fc4
Merge branch 'main' into fixes
breid-phys Jul 7, 2022
d8197d2
Merge pull request #1 from breid-phys/fixes
breid-phys Jul 7, 2022
bdb26bb
fixed index when multi-constellation
breid-phys Jul 8, 2022
82f8971
minor tweaks for high speed processing
breid-phys Sep 16, 2022
2ea8aca
force unique times
breid-phys Nov 4, 2022
0c6b3d2
actual bugfix
breid-phys Nov 4, 2022
bd3ad78
Merge pull request #2 from breid-phys/bugfix_time
breid-phys Nov 4, 2022
ef75b05
code formatting cleanup
breid-phys Dec 8, 2022
e7fd313
fast v3 loads to memory
breid-phys Dec 8, 2022
52b0e84
Merge branch 'main' into main
scivision Dec 27, 2022
c35576f
limited calls to genfromtxt for 2x speed
breid-phys Jan 18, 2023
b76b692
Merge pull request #3 from breid-phys/genfromtextfix
breid-phys Jan 18, 2023
5f50d9d
fixed datetime64 issue
breid-phys Jun 27, 2023
0d2f69b
fixed bug with nav errors
breid-phys Jun 29, 2023
9d28361
fixes for file reading edge cases
breid-phys Jun 29, 2023
6655124
added regex to reject bad potential sv
breid-phys Jul 5, 2023
a8579a2
fixed SBAS error for v2 fast code
breid-phys Oct 5, 2023
084e7b9
Merge branch 'main' into genfromtextfix
breid-phys Oct 5, 2023
548ede3
Merge pull request #4 from breid-phys/genfromtextfix
breid-phys Oct 5, 2023
bb1310d
added support for rinex v4
breid-phys Oct 17, 2023
7607aee
Merge pull request #5 from breid-phys/obs4
breid-phys Oct 17, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions channeldata.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"channeldata_version": 1,
"packages": {},
"subdirs": [
"noarch"
]
}
83 changes: 83 additions & 0 deletions index.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
<html>
<head>
<title>georinex</title>
<style type="text/css">
a, a:active {
text-decoration: none; color: blue;
}
a:visited {
color: #48468F;
}
a:hover, a:focus {
text-decoration: underline; color: red;
}
body {
background-color: #F5F5F5;
}
h2 {
margin-bottom: 12px;
}
th, td {
font: 100% monospace; text-align: left;
}
th {
font-weight: bold; padding-right: 14px; padding-bottom: 3px;
}
th.tight {
padding-right: 6px;
}
td {
padding-right: 14px;
}
td.tight {
padding-right: 8px;
}
td.s, th.s {
text-align: right;
}
td.summary {
white-space: nowrap;
overflow: hidden;
}
td.packagename {
white-space: nowrap;
text-overflow: ellipsis;
overflow: hidden;
max-width: 180px;
padding-right: 8px;
}
td.version {
//white-space: nowrap;
overflow: hidden;
max-width: 90px;
padding-right: 8px;
}
table {
background-color: white;
border-top: 1px solid #646464;
border-bottom: 1px solid #646464;
padding-top: 10px;
padding-bottom: 14px;
}
address {
color: #787878;
padding-top: 10px;
}
</style>
</head>
<body>
<h2>georinex</h2>
<h3><a href="rss.xml">RSS Feed</a>&nbsp;&nbsp;&nbsp;<a href="channeldata.json">channeldata.json</a></h3>
<a href="noarch">noarch</a>&nbsp;&nbsp;&nbsp; <table>
<tr>
<th style="padding-right:18px;">Package</th>
<th>Latest Version</th>
<th>Doc</th>
<th>Dev</th>
<th>License</th>
<th class="tight">noarch</th> <th>Summary</th>
</tr>
</table>
<address>Updated: 2023-06-08 14:39:19 +0000 - Files: 0</address>
</body>
</html>
9 changes: 9 additions & 0 deletions noarch/current_repodata.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"info": {
"subdir": "noarch"
},
"packages": {},
"packages.conda": {},
"removed": [],
"repodata_version": 1
}
Binary file added noarch/current_repodata.json.bz2
Binary file not shown.
82 changes: 82 additions & 0 deletions noarch/index.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
<html>
<head>
<title>georinex/noarch</title>
<style type="text/css">
a, a:active {
text-decoration: none; color: blue;
}
a:visited {
color: #48468F;
}
a:hover, a:focus {
text-decoration: underline; color: red;
}
body {
background-color: #F5F5F5;
}
h2 {
margin-bottom: 12px;
}
th, td {
font: 100% monospace; text-align: left;
}
th {
font-weight: bold; padding-right: 14px; padding-bottom: 3px;
}
td {
padding-right: 20px;
}
td.s, th.s {
text-align: right;
}
table {
background-color: white;
border-top: 1px solid #646464;
border-bottom: 1px solid #646464;
padding-top: 10px;
padding-bottom: 14px;
}
address {
color: #787878;
padding-top: 10px;
}
</style>
</head>
<body>
<h2>georinex/noarch</h2>
<table>
<tr>
<th>Filename</th>
<th>Size</th>
<th>Last Modified</th>
<th>SHA256</th>
<th>MD5</th>
</tr>
<tr>
<td><a href="repodata.json" alt="repodata.json">repodata.json</a></td>
<td class="s">127 B</td>
<td>2023-06-08 14:39:19 +0000</td>
<td>b546412dc20b790c5f9c223e394ff0e39a27ba12a99631ca4d1bb6c0ca3bd05c</td>
<td>917338f97423c09c911618661fef3056</td>
</tr> <tr>
<td><a href="repodata.json.bz2" alt="repodata.json.bz2">repodata.json.bz2</a></td>
<td class="s">126 B</td>
<td>2023-06-08 14:39:19 +0000</td>
<td>32d48c11b6d5ee61a54a9076f8c06a62ba795f69b671db3e44b1301fa0efae0e</td>
<td>1ecb5b301b9a2ba831f0a68a7ace5f00</td>
</tr> <tr>
<td><a href="repodata_from_packages.json" alt="repodata_from_packages.json">repodata_from_packages.json</a></td>
<td class="s">127 B</td>
<td>2023-06-08 14:39:19 +0000</td>
<td>b546412dc20b790c5f9c223e394ff0e39a27ba12a99631ca4d1bb6c0ca3bd05c</td>
<td>917338f97423c09c911618661fef3056</td>
</tr> <tr>
<td><a href="repodata_from_packages.json.bz2" alt="repodata_from_packages.json.bz2">repodata_from_packages.json.bz2</a></td>
<td class="s">126 B</td>
<td>2023-06-08 14:39:19 +0000</td>
<td>32d48c11b6d5ee61a54a9076f8c06a62ba795f69b671db3e44b1301fa0efae0e</td>
<td>1ecb5b301b9a2ba831f0a68a7ace5f00</td>
</tr> </table>
<address>Updated: 2023-06-08 14:39:19 +0000 - Files: 0</address>
</body>
</html>
9 changes: 9 additions & 0 deletions noarch/repodata.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"info": {
"subdir": "noarch"
},
"packages": {},
"packages.conda": {},
"removed": [],
"repodata_version": 1
}
Binary file added noarch/repodata.json.bz2
Binary file not shown.
9 changes: 9 additions & 0 deletions noarch/repodata_from_packages.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"info": {
"subdir": "noarch"
},
"packages": {},
"packages.conda": {},
"removed": [],
"repodata_version": 1
}
Binary file added noarch/repodata_from_packages.json.bz2
Binary file not shown.
18 changes: 15 additions & 3 deletions src/georinex/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
from .rio import rinexinfo
from .obs2 import rinexobs2
from .obs3 import rinexobs3
from .obs4 import rinexobs4
from .nav2 import rinexnav2
from .nav3 import rinexnav3
from .sp3 import load_sp3
Expand Down Expand Up @@ -154,7 +155,7 @@ def rinexnav(

if fn.suffix == ".nc":
try:
return xarray.open_dataset(fn, group=group)
return xarray.load_dataset(fn, group=group)
except OSError as e:
raise LookupError(f"Group {group} not found in {fn} {e}")

Expand Down Expand Up @@ -205,7 +206,7 @@ def rinexobs(
# %% NetCDF4
if fn.suffix == ".nc":
try:
return xarray.open_dataset(fn, group=group)
return xarray.load_dataset(fn, group=group)
except OSError as e:
raise LookupError(f"Group {group} not found in {fn} {e}")

Expand Down Expand Up @@ -235,6 +236,17 @@ def rinexobs(
fast=fast,
interval=interval,
)
elif int(info["version"]) == 4:
obs = rinexobs4(
fn,
use,
tlim=tlim,
useindicators=useindicators,
meas=meas,
verbose=verbose,
fast=fast,
interval=interval,
)
else:
raise ValueError(f"unknown RINEX {info} {fn}")

Expand All @@ -259,7 +271,7 @@ def _groupexists(fn: Path, group: str, overwrite: bool) -> str:

# be sure there isn't already NAV in it
try:
xarray.open_dataset(fn, group=group)
xarray.load_dataset(fn, group=group)
raise ValueError(f"{group} already in {fn}")
except OSError:
pass
Expand Down
28 changes: 16 additions & 12 deletions src/georinex/obs2.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,18 +28,17 @@ def rinexobs2(
*,
fast: bool = True,
interval: float | int | timedelta = None,
):
) -> xarray.Dataset:

if isinstance(use, str):
use = {use}

if not use:
use = {"C", "E", "G", "J", "R", "S"}

obs = xarray.Dataset(
{}, coords={"time": np.array([], dtype="datetime64[ns]"), "sv": np.array([], dtype="<U3")}
)
attrs: dict[T.Hashable, T.Any] = {}
obs = xarray.Dataset({}, coords={"time": np.array([], dtype="datetime64[ns]"), "sv": []})
attrs: dict[str, T.Any] = {}

for u in use:
o = rinexsystem2(
fn,
Expand Down Expand Up @@ -236,7 +235,10 @@ def rinexsystem2(
assert darr.shape[0] == gsv.size

# %% select only "used" satellites
isv = [int(s[1:]) - 1 for s in gsv]
if system == "S":
isv = range(len(gsv))
else:
isv = [int(s[1:]) - 1 for s in gsv]

for i, k in enumerate(hdr["fields_ind"]):
if useindicators:
Expand Down Expand Up @@ -266,17 +268,20 @@ def rinexsystem2(
else:
fields.extend([None, None])

obs = xarray.Dataset(
coords={"time": times, "sv": [f"{system}{i:02d}" for i in range(1, Nsvsys + 1)]}
)
if system == "S":
obs = xarray.Dataset(coords={"time": times, "sv": gsv})
else:
obs = xarray.Dataset(
coords={"time": times, "sv": [f"{system}{i:02d}" for i in range(1, Nsvsys + 1)]}
)

for i, k in enumerate(fields):
# FIXME: for limited time span reads, this drops unused data variables
# if np.isnan(data[i, ...]).all():
# continue
if k is None:
continue
obs[k] = (("time", "sv"), data[i, :, :])
obs[k] = (("time", "sv"), data[i, :, : len(obs["sv"])])

obs = obs.dropna(dim="sv", how="all")
obs = obs.dropna(dim="time", how="all") # when tlim specified
Expand Down Expand Up @@ -391,7 +396,7 @@ def obsheader2(
hdr["Nl_sv"] = ceil(hdr["Nobs"] / 5)
# %% list with receiver location in x,y,z cartesian ECEF (OPTIONAL)
try:
hdr["position"] = [float(j) for j in hdr["APPROX POSITION XYZ"].split()]
hdr["position"] = [float(j) for j in hdr["APPROX POSITION XYZ"].split()][:3]
if ecef2geodetic is not None:
hdr["position_geodetic"] = ecef2geodetic(*hdr["position"])
except (KeyError, ValueError):
Expand Down Expand Up @@ -561,7 +566,6 @@ def _timehdr(ln: str) -> datetime:


def _timeobs(ln: str) -> datetime:

year = int(ln[1:3])
if year < 80:
year += 2000
Expand Down
Loading