Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Land Obstacles - 278 #436

Open
wants to merge 98 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 87 commits
Commits
Show all changes
98 commits
Select commit Hold shift + click to select a range
c4621ee
first commit of land mass etl pipline
SPDonaghy Mar 12, 2024
8d0cb36
script test passed
SPDonaghy Mar 13, 2024
e76f0aa
implemented autodownload of missing files
SPDonaghy Mar 14, 2024
7cf0fd6
added command to install requirements
SPDonaghy Mar 14, 2024
c1fa206
deactivate script until polygonization checks are implemented
SPDonaghy Mar 15, 2024
492aeea
added polygonization check per chunk
SPDonaghy Mar 16, 2024
3fa5716
removed pip install command because imports come first anyway
SPDonaghy Mar 16, 2024
3ee06e1
clarified which section is skipped with --jump
SPDonaghy Mar 16, 2024
9144277
added file io funcs to separate file
SPDonaghy Mar 16, 2024
fb08b57
added logger after all
SPDonaghy Mar 16, 2024
26255a9
write sindex file even in test mode
SPDonaghy Mar 17, 2024
635d9fc
created land class
SPDonaghy Mar 17, 2024
111aa0e
Create directory to download zip to if it dne
patrick-5546 Mar 19, 2024
2762a88
Merge remote-tracking branch 'origin/main' into user/SPDonaghy/implem…
patrick-5546 Mar 19, 2024
32984d8
Do the same for dumping pkls
patrick-5546 Mar 19, 2024
34897ea
moved logger defn to files.py
SPDonaghy Mar 19, 2024
0aca880
Add missing dependency
patrick-5546 Mar 20, 2024
63cd01b
more land class implementation
SPDonaghy Mar 23, 2024
dfd20d7
Merge branch 'user/SPDonaghy/implement-land-obstacles-278' of https:/…
SPDonaghy Mar 23, 2024
005104b
.
SPDonaghy Mar 23, 2024
77e882a
typo
SPDonaghy Mar 23, 2024
a3184e0
rm envelope func call
SPDonaghy Apr 15, 2024
1c90341
Merge branch 'main' into user/SPDonaghy/implement-land-obstacles-278
SPDonaghy Jun 1, 2024
1aa064d
moved utils to main file
SPDonaghy Jun 1, 2024
c6b07f6
removed utils file
SPDonaghy Jul 3, 2024
b35e592
refactored land functions, no logic changes
SPDonaghy Jul 3, 2024
61123a0
removed bathymetric data processing from etl script
SPDonaghy Jul 3, 2024
287e185
added dirs for data files
SPDonaghy Jul 3, 2024
705a648
added fiona requirement
SPDonaghy Jul 3, 2024
12e40b5
buffer is added during data processing instead of runtime
SPDonaghy Jul 3, 2024
8b4e894
removed buffering for land from obstacles class
SPDonaghy Jul 3, 2024
cf5dbcd
added test notebook file
SPDonaghy Jul 3, 2024
505689e
updated package.xml and some comments
SPDonaghy Jul 3, 2024
d374476
add geopandas and fiona to container
SPDonaghy Sep 14, 2024
eea93b5
renamed some attrs for Land
SPDonaghy Sep 14, 2024
0433395
fixed transform polygons name error
SPDonaghy Sep 14, 2024
8a91c45
added tqdm progress bar back to etl
SPDonaghy Sep 14, 2024
b942822
update gitignore to not ignore shp and pkl dirs
SPDonaghy Sep 15, 2024
6bd4f58
updated giginore
SPDonaghy Sep 15, 2024
6c57da5
gitignore trickery
SPDonaghy Sep 15, 2024
fe9ef43
un gitignore datafiles
SPDonaghy Sep 15, 2024
f9cd58a
added binary files, sorry git
SPDonaghy Sep 15, 2024
d7a49f7
fixed wrong types for some variables
SPDonaghy Sep 15, 2024
1579046
some tests added
SPDonaghy Sep 15, 2024
bb2e868
initial implementation of non-visual tests
SPDonaghy Sep 16, 2024
2e5e5a7
small fixes to tests
SPDonaghy Sep 16, 2024
dd36e38
Merge branch 'main' into user/SPDonaghy/implement-land-obstacles-278
SPDonaghy Sep 16, 2024
7beab18
all tests pass, removed scrap notebook
SPDonaghy Sep 16, 2024
5108db8
moved visual tests to notebook
SPDonaghy Sep 16, 2024
1c4a101
completed visual tests
SPDonaghy Sep 16, 2024
01eeecc
fixed mypy issues
SPDonaghy Sep 16, 2024
e7b6941
updated notebook
SPDonaghy Sep 16, 2024
8d472c5
added tqdm to docker image
SPDonaghy Sep 16, 2024
05a7464
resolve colcon test issue
SPDonaghy Sep 16, 2024
e25b971
trying to fix colcon
SPDonaghy Sep 16, 2024
2199ff6
debugging colcon
SPDonaghy Sep 16, 2024
cb28f0e
please
SPDonaghy Sep 16, 2024
8ed0c2c
python3
SPDonaghy Sep 16, 2024
aaf9108
added fix to test.yml
SPDonaghy Sep 16, 2024
5ae25a9
rm python CI
SPDonaghy Sep 16, 2024
c632030
pls
SPDonaghy Sep 16, 2024
3c6476e
.
SPDonaghy Sep 16, 2024
e5ae6eb
another test
SPDonaghy Sep 16, 2024
464cd96
pin numpy
SPDonaghy Sep 16, 2024
bb22901
visualize bounding boxes in notebook
SPDonaghy Sep 16, 2024
6d2e098
woops
SPDonaghy Sep 16, 2024
d2ac8b1
land multipolygon is now clipped by bounding box
SPDonaghy Sep 16, 2024
5f8ae49
added some calcs to the notebook
SPDonaghy Sep 16, 2024
8354c05
renamed land data generator script
SPDonaghy Sep 18, 2024
9342d1f
added tqdm to pkg xml
SPDonaghy Sep 18, 2024
f21c07e
just some comment changes
SPDonaghy Sep 21, 2024
6bab848
simplified land class
SPDonaghy Sep 25, 2024
26af172
Merge branch 'main' into user/SPDonaghy/implement-land-obstacles-278
SPDonaghy Sep 25, 2024
37a5d4b
remove geopandas again
SPDonaghy Sep 25, 2024
178af8c
updated tests
SPDonaghy Sep 25, 2024
a0fa1f1
updated notebook with visual tests
SPDonaghy Sep 25, 2024
bad1e5e
updated state points in the visual
SPDonaghy Sep 26, 2024
cb6a9f4
metadata
SPDonaghy Sep 26, 2024
dce9ad2
modified workflow to pip3 list
SPDonaghy Sep 26, 2024
38f8ef4
removed union_all to maintain compatibility with shapely 1.8.0
SPDonaghy Sep 26, 2024
086d246
unit tests pass
SPDonaghy Sep 26, 2024
fb2260a
test custom bbox
SPDonaghy Sep 27, 2024
5f4c8d1
updated notebook for custom state space
SPDonaghy Sep 27, 2024
c434687
renamed bbox to statespace
SPDonaghy Sep 27, 2024
3576250
rolled back shapely refs to 1.8.0
SPDonaghy Sep 27, 2024
3145a2b
updated comments
SPDonaghy Sep 27, 2024
099c565
ensured that visual plots look correct
SPDonaghy Sep 28, 2024
f080ae5
renamed func to _latlon_polygon_list_to_xy_polygon_list
SPDonaghy Oct 29, 2024
3bc5933
added docstring
SPDonaghy Oct 29, 2024
8d2c737
rename constant
SPDonaghy Oct 29, 2024
494be72
updated notebook, still works
SPDonaghy Oct 29, 2024
f660ec4
simplified polygon transforming function
SPDonaghy Oct 29, 2024
0c7bc4d
updated docstring
SPDonaghy Oct 29, 2024
c08a495
rm redundant check for type in update ref pt
SPDonaghy Oct 29, 2024
8d33d27
move latlon to xy poly function to coord systems
SPDonaghy Oct 29, 2024
f21c350
some refactoring, bad message sorry
SPDonaghy Oct 29, 2024
7ef0c64
updates to test file
SPDonaghy Oct 29, 2024
e4e05a8
updated test and docstring
SPDonaghy Oct 29, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .devcontainer/base-dev/base-dev.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -374,4 +374,4 @@ ENV DEBIAN_FRONTEND=
# install dev python3 dependencies
RUN pip3 install \
# for juypter notebooks
ipykernel
ipykernel \
9 changes: 9 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,15 @@ jobs:
- name: Checkout workspace
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v3
with:
python-version: '3.x'

- name: show installed python dependencies
run: |
pip3 list

- name: Test
uses: ./.github/actions/
with:
Expand Down
70 changes: 70 additions & 0 deletions src/local_pathfinding/land/generate_land_data.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
import csv
import os
import pickle
from os.path import normpath
from typing import Any

import geopandas as gpd
import pyproj
from shapely.geometry import LineString, MultiPolygon, Polygon
from shapely.ops import split


def dump_pkl(object: Any, file_path: str):
os.makedirs(os.path.dirname(file_path), exist_ok=True)
with open(file_path, "wb") as f:
pickle.dump(object, f, protocol=pickle.HIGHEST_PROTOCOL)


def load_pkl(file_path: str) -> Any:
with open(file_path, "rb") as f:
return pickle.load(f)


def pickle_land():
SPDonaghy marked this conversation as resolved.
Show resolved Hide resolved

WGS84 = pyproj.CRS("EPSG:4326")
MERCATOR = pyproj.CRS("EPSG:3857")
INLAND_POLYGON = normpath(
"/workspaces/sailbot_workspace/src/local_pathfinding/land/csv/map_sel.csv"
)
BUFFER_AMOUNT = 5000 # meters

gdf = gpd.read_file(
"/workspaces/sailbot_workspace/src/local_pathfinding/land/territorial_waters.geojson"
)

# the only useful data is in the id and geometry columns and in the first two rows
gdf = gdf[["id", "geometry"]]
gdf = gdf.head(2)
geoms = list(gdf.iloc[0]["geometry"].geoms) + list(gdf.iloc[1]["geometry"].geoms)

# convert to MultiPolygon for easier geometric manipulation
multi_poly = MultiPolygon(geoms)
multi_poly = multi_poly.buffer(0) # repairs invalid geometry
# this is to prevent geometry the crosses -180 longitude from being distorted by transforms
with open(INLAND_POLYGON, "r") as f:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could add error handling here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! which errors do you think we should plan for here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file cannot open exception error

reader = csv.reader(f)
reader.__next__()
map_sel = Polygon([[float(row[1]), float(row[0])] for row in reader])
IDL = LineString([(-179.5, 90), (-179.5, -90)])
map_sel_east = split(map_sel, IDL).geoms[0]
multi_poly_clipped = multi_poly.intersection(map_sel_east)

# back to gdf for easier coordinate system transformation
gdf_clipped = gpd.GeoDataFrame(geometry=[multi_poly_clipped], crs=WGS84)
gdf_clipped.to_crs(MERCATOR, inplace=True)
gdf_clipped["geometry"] = gdf_clipped.buffer(BUFFER_AMOUNT)
gdf_clipped.to_crs(WGS84, inplace=True)

# Land class will deal with Multipolygon
multi_poly_clipped_buffered = MultiPolygon(list(gdf_clipped.iloc[0]["geometry"].geoms))

dump_pkl(
multi_poly_clipped_buffered,
"/workspaces/sailbot_workspace/src/local_pathfinding/land/pkl/land.pkl",
)


if __name__ == "__main__":
pickle_land()
11,975 changes: 11,784 additions & 191 deletions src/local_pathfinding/land/land_polygons_notebook.ipynb

Large diffs are not rendered by default.

Binary file modified src/local_pathfinding/land/pkl/land.pkl
Binary file not shown.
Loading
Loading