Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up example notebooks #210

Merged
merged 3 commits into from
Dec 19, 2023
Merged

Clean up example notebooks #210

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f8c026a
Rename notebooks; move data files
brynpickering Dec 5, 2023
adc6190
Rename notebooks; move wiki gen script
brynpickering Dec 19, 2023
ba83b7f
Merge branch 'main' into clean-example-notebooks
brynpickering Dec 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 3 additions & 12 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ coverage.xml
*.cover
.hypothesis/
.pytest_cache/
reports/

# Translations
*.mo
Expand Down Expand Up @@ -122,15 +123,5 @@ dmypy.json
.pyre/
.idea

tests/test_outputs/
test_chunks.csv
reports/
mprofile*
benchmark-outputs/
working-dir/
scripts/secrets-mgr-exploration.py
*.nbconvert.*
example_data/pt2matsim_network/genet_output/*
genet_output/*
example_data/output_*/*
example_data/api_requests_send.json
# Project specific
examples/example_data/outputs
9 changes: 5 additions & 4 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,14 +65,15 @@ automated `PEP8` checks and runs unit tests in a fresh environment, as well as i
8. Update/add to or generate a new jupyter notebook in `notebooks` directory which takes the user through your new feature or
change.
1. Jupyter notebooks are closely linked to the [wiki pages](https://github.com/arup-group/genet/wiki).
1. Make sure you follow the naming convention: `number.number. Theme: Catchy Title`
2. Make sure you structure the notebook in a way that you would like to see it in a wiki page, with a lot of
1. Make sure you follow the naming convention to title your page: `number.number. Theme: Catchy Title`
2. Make sure to also name the notebook so that it is machine readable: `number_number_theme_catchy_title.ipynb`
3. Make sure you structure the notebook in a way that you would like to see it in a wiki page, with a lot of
markdown cells containing quality descriptions. Use existing notebook as examples.
2. After your changes have been merged, you may like to update the wiki pages. To do this in an automated way:
1. Clone the wiki part of the repo: ```git clone https://github.com/arup-group/genet.wiki.git```
2. Run the `notebooks/generate_usage_wiki_from_notebooks.py` script, pointing at the folder containing the
2. Run the `examples/generate_usage_wiki_from_notebooks.py` script, pointing at the folder containing the
notebooks and the wiki repo folder to receive the output.
3. You may use example data already in `example_data` directory of this repo, or add more (small amount of) data to
3. You may use example data already in `examples/example_data` directory of this repo, or add more (small amount of) data to
it to show off your new features.
9. Add section in the `README.md` which shows usage of your new feature. This can be paraphrased from the jupyter
notebook in point above.
Expand Down
18 changes: 9 additions & 9 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,25 +192,25 @@ We welcome community contributions to GeNet; please see our [guide to contributi
described below to verify that the code still works. All of the following commands assume you are in the project's root
directory.

### Unit tests
To run unit tests within genet python environment:
### tests
To run unit tests within genet python environment (including testing example notebooks):

python -m pytest -vv tests
pytest

and within a docker container:

docker run cml-genet pytest -vv tests
docker run cml-genet pytest -vv

In either case, for shorter test runtimes, append 'tests/' to limit to the unit tests and ignore notebook tests.

### Generate a unit test code coverage report

To generate an HTML coverage report at `reports/coverage/index.html`:

./bash_scripts/code-coverage.sh
pytest --cov-report=html

Coverage will also be tracked in pull requests.

### Lint the python code

Run `pre-commit install` to install pre-commit, which will lint and format your code whenever you commit staged changes.

### Smoke test the Jupyter notebooks

./bash_scripts/notebooks-smoke-test.sh
1 change: 0 additions & 1 deletion example_data/api_requests_send.json
View file
Open in desktop

This file was deleted.

6 changes: 3 additions & 3 deletions notebooks/1. Intro.ipynb → examples/1_intro_to_genet.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@
}
},
"source": [
"# Intro\n",
"# 1. Introduction to GeNet\n",
"This section goes through basic usage examples. Available as a jupyter notebook and a wiki page.\n",
"\n",
KasiaKoz marked this conversation as resolved.
Show resolved Hide resolved
"## `Network` data structure\n",
"\n",
"![GeNet Network Graph diagram](https://github.com/arup-group/genet/blob/master/images/genet_network_graph.png)\n",
"![GeNet Network Graph diagram](../images/genet_network_graph.png)\n",
"\n",
"Main schema:\n",
"* nodes: exist uniquely, hold spatial information.\n",
Expand All @@ -28,7 +28,7 @@
"* nodes hold spatial information\n",
"* each edge/link holds information such as mode of transport allowed, speed, capacity, length, OSM-inherited tags\n",
"\n",
"![GeNet Schedule Graph diagram](https://github.com/arup-group/genet/blob/master/images/genet_schedule.png)\n",
"![GeNet Schedule Graph diagram](../images/genet_schedule.png)\n",
"\n",
"`Schedule` consists of:\n",
"* List of `Service`s, where each `Service` has:\n",
Expand Down
4 changes: 2 additions & 2 deletions notebooks/2.1. Reading Data: MATSim.ipynb → examples/2_1_reading_data_matsim.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Reading a MATSim network\n",
"# 2.1. Reading data: MATSim networks\n",
"\n",
"This page goes through methods for reading in MATSim networks. Available as a jupyter notebook or a wiki page."
]
Expand Down Expand Up @@ -37,7 +37,7 @@
},
"outputs": [],
"source": [
"path_to_matsim_network = \"../example_data/pt2matsim_network\"\n",
"path_to_matsim_network = \"example_data/pt2matsim_network\"\n",
"\n",
"network = os.path.join(path_to_matsim_network, \"network.xml\")\n",
"schedule = os.path.join(path_to_matsim_network, \"schedule.xml\")\n",
Expand Down
6 changes: 3 additions & 3 deletions notebooks/2.2. Reading Data: OSM.ipynb → examples/2_2_reading_data_osm.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Reading OSM data\n",
"# 2.2. Reading data: OSM\n",
"\n",
"Reading OSM data can be configured. You will find example config in the configs directory. Available as a jupyter notebook or wiki page.\n",
"\n",
Expand Down Expand Up @@ -158,7 +158,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"2022-07-14 15:27:55,774 - Building OSM graph from file ../example_data/example.osm\n",
"2022-07-14 15:27:55,774 - Building OSM graph from file example_data/example.osm\n",
"2022-07-14 15:27:56,540 - Creating networkx graph from OSM data\n",
"2022-07-14 15:27:56,541 - OSM: Extract Nodes and Paths from OSM data\n",
"2022-07-14 15:27:56,898 - OSM: Add each OSM way (aka, path) to the OSM graph\n",
Expand All @@ -176,7 +176,7 @@
],
"source": [
"n = read_osm(\n",
" \"../example_data/example.osm\",\n",
" \"example_data/example.osm\",\n",
" \"../genet/configs/OSM/slim_config.yml\",\n",
" num_processes=2,\n",
" epsg=\"epsg:27700\",\n",
Expand Down
15 changes: 7 additions & 8 deletions notebooks/2.3. Reading Data: GTFS.ipynb → examples/2_3_reading_data_gtfs.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,10 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Reading GTFS data\n",
"# 2.3. Reading data: GTFS\n",
"\n",
"This page goes through methods for reading in GTFS ([General Transit Feed Specification](https://developers.google.com/transit/gtfs)). [Here](https://developers.google.com/transit/gtfs/reference) is the reference page for the schema of GTFS data. Available as a jupyter notebook or wiki page.\n",
"\n",
"Small sample of this data can be found in [tests/test_data/gtfs](https://github.com/arup-group/genet/tree/master/tests/test_data/gtfs).\n",
"This page goes through methods for reading in GTFS ([General Transit Feed Specification](https://developers.google.com/transit/gtfs)). \n",
"[Here](https://developers.google.com/transit/gtfs/reference) is the reference page for the schema of GTFS data.\n",
"\n",
"GeNet ingests zipped or unzipped GTFS feeds. The following files are required in the unzipped folder, or inside the zip file:\n",
"- `calendar.txt`\n",
Expand All @@ -21,8 +20,8 @@
"\n",
"GeNet does support extraction of services/routes/stops in the output `genet.Schedule` object based on a geographical area (Methods: `services_on_spatial_condition`, `routes_on_spatial_condition`, `stops_on_spatial_condition`, more information in notebook on using genet Network), but you might like to use [gtfs-lib](https://github.com/conveyal/gtfs-lib) prior to ingestion in GeNet.\n",
"\n",
"The user assumes responsibility for the quality of their input GTFS feed. There are various validation tools that can \n",
"be used with GTFS feeds before using with GeNet, see this page for a summary of [validation tools](https://gtfs.org/testing/)."
"The user assumes responsibility for the quality of their input GTFS feed. \n",
"There are various validation tools that can be used with GTFS feeds before using with GeNet, see [this page](https://gtfs.org/testing/) for a summary of validation tools."
]
},
{
Expand Down Expand Up @@ -60,7 +59,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"2022-07-14 15:27:50,737 - Reading GTFS from ../example_data/example_gtfs\n",
"2022-07-14 15:27:50,737 - Reading GTFS from example_data/example_gtfs\n",
"2022-07-14 15:27:50,740 - Reading the calendar for GTFS\n",
"2022-07-14 15:27:50,743 - Reading GTFS data into usable format\n",
"2022-07-14 15:27:50,745 - Reading stop times\n",
Expand All @@ -71,7 +70,7 @@
}
],
"source": [
"s = read_gtfs(\"../example_data/example_gtfs\", \"20190603\")"
"s = read_gtfs(\"example_data/example_gtfs\", \"20190603\")"
]
},
{
Expand Down
10 changes: 5 additions & 5 deletions notebooks/2.4. Reading Data: CSV.ipynb → examples/2_4_reading_data_csv.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Reading a network from CSV\n",
"# 2.4. Reading data: CSV\n",
"Available as a jupyter notebook or wiki page.\n",
"\n",
"You can read a network from node and link CSV files.\n",
Expand Down Expand Up @@ -39,8 +39,8 @@
},
"outputs": [],
"source": [
"csv_nodes = \"../example_data/example_csv/nodes.csv\"\n",
"csv_links = \"../example_data/example_csv/links.csv\""
"csv_nodes = \"example_data/example_csv/nodes.csv\"\n",
"csv_links = \"example_data/example_csv/links.csv\""
]
},
{
Expand All @@ -57,8 +57,8 @@
"name": "stderr",
"output_type": "stream",
"text": [
"2022-07-14 15:30:48,943 - Reading nodes from ../example_data/example_csv/nodes.csv\n",
"2022-07-14 15:30:48,949 - Reading links from ../example_data/example_csv/nodes.csv\n",
"2022-07-14 15:30:48,943 - Reading nodes from example_data/example_csv/nodes.csv\n",
"2022-07-14 15:30:48,949 - Reading links from example_data/example_csv/nodes.csv\n",
"2022-07-14 15:30:49,024 - Added 4 nodes\n",
"2022-07-14 15:30:49,034 - Added 2 links\n"
]
Expand Down
26 changes: 13 additions & 13 deletions ...s/2.5. Reading Data: JSON & GeoJSON.ipynb → examples/2_5_reading_data_json_geojson.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Reading JSON & GeoJSON data\n",
"# 2.5. Reading data: JSON & GeoJSON\n",
"Available as a jupyter notebook or wiki page.\n",
"\n",
"GeNet lets you read JSON data into a Networn graph and PT Schedule, or GeoJSON into Network Graph. \n",
Expand Down Expand Up @@ -74,14 +74,14 @@
"name": "stderr",
"output_type": "stream",
"text": [
"2022-07-14 15:31:04,968 - Reading Network from ../example_data/example_json/network.json\n",
"2022-07-14 15:31:04,968 - Reading Network from example_data/example_json/network.json\n",
"2022-07-14 15:31:05,058 - Added 4 nodes\n",
"2022-07-14 15:31:05,067 - Added 2 links\n"
]
}
],
"source": [
"n = read_json_network(\"../example_data/example_json/network.json\", \"epsg:27700\")"
"n = read_json_network(\"example_data/example_json/network.json\", \"epsg:27700\")"
]
},
{
Expand Down Expand Up @@ -211,12 +211,12 @@
"name": "stderr",
"output_type": "stream",
"text": [
"2022-07-14 15:31:05,075 - Reading Schedule from ../example_data/example_json/schedule.json\n"
"2022-07-14 15:31:05,075 - Reading Schedule from example_data/example_json/schedule.json\n"
]
}
],
"source": [
"s = read_json_schedule(\"../example_data/example_json/schedule.json\", \"epsg:27700\")"
"s = read_json_schedule(\"example_data/example_json/schedule.json\", \"epsg:27700\")"
]
},
{
Expand All @@ -240,17 +240,17 @@
"name": "stderr",
"output_type": "stream",
"text": [
"2022-07-14 15:31:05,119 - Reading Network from ../example_data/example_json/network.json\n",
"2022-07-14 15:31:05,119 - Reading Network from example_data/example_json/network.json\n",
"2022-07-14 15:31:05,281 - Added 4 nodes\n",
"2022-07-14 15:31:05,354 - Added 2 links\n",
"2022-07-14 15:31:05,360 - Reading Schedule from ../example_data/example_json/schedule.json\n"
"2022-07-14 15:31:05,360 - Reading Schedule from example_data/example_json/schedule.json\n"
]
}
],
"source": [
"n = read_json(\n",
" network_path=\"../example_data/example_json/network.json\",\n",
" schedule_path=\"../example_data/example_json/schedule.json\",\n",
" network_path=\"example_data/example_json/network.json\",\n",
" schedule_path=\"example_data/example_json/schedule.json\",\n",
" epsg=\"epsg:27700\",\n",
")"
]
Expand Down Expand Up @@ -307,8 +307,8 @@
"name": "stderr",
"output_type": "stream",
"text": [
"2022-07-14 15:31:05,419 - Reading Network nodes from ../example_data/example_geojson/network_nodes.geojson\n",
"2022-07-14 15:31:05,520 - Reading Network links from ../example_data/example_geojson/network_links.geojson\n",
"2022-07-14 15:31:05,419 - Reading Network nodes from example_data/example_geojson/network_nodes.geojson\n",
"2022-07-14 15:31:05,520 - Reading Network links from example_data/example_geojson/network_links.geojson\n",
"2022-07-14 15:31:05,613 - Added 4 nodes\n",
"2022-07-14 15:31:05,622 - Added 2 links\n"
]
Expand All @@ -318,8 +318,8 @@
"from genet import read_geojson_network\n",
"\n",
"n = read_geojson_network(\n",
" \"../example_data/example_geojson/network_nodes.geojson\",\n",
" \"../example_data/example_geojson/network_links.geojson\",\n",
" \"example_data/example_geojson/network_nodes.geojson\",\n",
" \"example_data/example_geojson/network_links.geojson\",\n",
" \"epsg:27700\",\n",
")"
]
Expand Down
17 changes: 9 additions & 8 deletions notebooks/3.1. Writing Data: MATSim.ipynb → examples/3_1_writing_data_matsim.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Writing `Network` to MATSim\n",
"# 3.1. Writing data: MATSim\n",
"\n",
"This page goes through the MATSim files that can be generated with GeNet. Available as a jupyter notebook or wiki page.\n",
"\n",
Expand Down Expand Up @@ -72,7 +72,8 @@
"\n",
"from genet import read_matsim\n",
"\n",
"path_to_matsim_network = \"../example_data/pt2matsim_network\"\n",
"path_to_matsim_network = \"example_data/pt2matsim_network\"\n",
"path_to_output = \"example_data/outputs/pt2matsim_network\"\n",
"\n",
"network = os.path.join(path_to_matsim_network, \"network.xml\")\n",
"schedule = os.path.join(path_to_matsim_network, \"schedule.xml\")\n",
Expand Down Expand Up @@ -107,14 +108,14 @@
"name": "stderr",
"output_type": "stream",
"text": [
"2022-07-14 15:31:43,952 - Writing ../example_data/pt2matsim_network/genet_output/network.xml\n",
"2022-07-14 15:31:44,529 - Writing ../example_data/pt2matsim_network/genet_output/schedule.xml\n",
"2022-07-14 15:31:44,570 - Writing ../example_data/pt2matsim_network/genet_output/vehicles.xml\n"
"2022-07-14 15:31:43,952 - Writing example_data/pt2matsim_network/genet_output/network.xml\n",
"2022-07-14 15:31:44,529 - Writing example_data/pt2matsim_network/genet_output/schedule.xml\n",
"2022-07-14 15:31:44,570 - Writing example_data/pt2matsim_network/genet_output/vehicles.xml\n"
]
}
],
"source": [
"n.write_to_matsim(os.path.join(path_to_matsim_network, \"genet_output\"))"
"n.write_to_matsim(path_to_output)"
]
},
{
Expand All @@ -129,7 +130,7 @@
"kernelspec": {
"display_name": "genet",
"language": "python",
"name": "genet"
"name": "python3"
},
"language_info": {
"codemirror_mode": {
Expand All @@ -141,7 +142,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
"version": "3.11.5"
},
"latex_envs": {
"LaTeX_envs_menu_present": true,
Expand Down
Loading
Loading