08 add map #9

crispy-wonton · 2023-10-17T09:04:57Z

Fixes #8

Changes:

add kepler map code to repo and all the supporting functions including utils.py file for using Arial font
add documentation of data versions for October 2023 work to README

Instructions for Reviewer

please could you checkout the branch and run the code as instructed by the README and test that you get hp_map.html produced and it looks correct

Checklist:

improve and correct set up instructions and update output file extensions to html.

update local_data_dir to use relative file path from config base.yaml so config file can be updated by users rather than editing scripts.

change output graphs from produce_plots.py from .png to .html files to avoid altair CalledProcessError

- add argparser to allow user to specify local_data_dir, epc-, and mcs-batch - add function to load MCS data directly from S3 bucket - add function to check for EPC data locally and download from S3 if not located - update existing functions to work with above - add global parameters for EPC processing version to base.yaml config file

update instructions in README to explain new way of running script and to record batches for historical analyses

add script to generate stats and charts from asf_senedd_response into produce_plots.py

merge getters from loading.py in asf_senedd_response into get_data.py script

update EPC getter function name to match updates in get_data.py

add files: - translation_config for welsh translations - plotting.py for plotting functions - augmenting.py for data processing and augmentation

…repo

update to prevent module not subscriptable error

improve selection of min and max dates

allows user to specify which directory with supplementary data to use, meaning new supplementary data can be added to analysis as it's updated

…_add_map

sofiapinto

hey @crispy-wonton ,

thanks for this PR - the code for the map looks good. I didn't look to closely at the specific Kepler function calls, but overall looks like it's doing what is supposed to. Also haven't tested if the numbers make sense as in, if areas of higher percentage are correct.

When I open the HTML it shows San Francisco are. I had to zoom out and then look for wales. There my be a way of centering the map in wales. Something like:

wales_map = KeplerGl(...)
wales_map.config = {
    'version': 'v1',
    'config': {
        'mapState': {
            'latitude': xxxx,  # Specific latitude value
            'longitude':xxx,  # Specific longitude value
            'zoom': 10  # Specify the desired zoom level
        }
    }
}

I also gave a suggestion of how to re-define the ranges in the color scale, but haven't tested it so i'm not sure it works.

The code doesn't run as is, given the changes done in the other branch, so I had to change a few things around and comment parts of the code so that the map would be generated, but i managed to generate it after those changes. I'm happy to re-run the code to make sure if runs smoothly after you incorporate changes from main (after merging the other branch).

sofiapinto · 2023-10-25T11:00:42Z

README.md

+- You can specify which batch of EPC data to download and MCS data to load from S3 by passing the `--epc_batch` and `--mcs_batch` arguments, both
+  default to downloading/loading the newest data from S3, respectively. You can also specify which set of supplementary data should be used by passing
+  the `--supp_data` argument followed by the name of the directory, e.g. data_202310. See the `Historical analyses` section below to see which version was used for each analysis.
+  Run `python asf_welsh_energy_consultation/analysis/produce_plots.py -h` for more info.


This will need to be changed after you merge the PR number 3, as it's now produce_plots_and_stats.py

sofiapinto · 2023-10-25T11:01:28Z

README.md

+October 2023 analysis:
+
+- Supplementary data: data_202310
+- EPC: 2023_Q2_complete (preprocessed)
+- mcs_installations_231009.csv
+- mcs_installations_epc_full_231009.csv
+- off-gas-live-postcodes-2022.xlsx - check [here](https://www.xoserve.com/a-to-z/) for updates
+- rurality.ods - 2011 Rural Urban Classification for small area geographies, see [here](https://www.ons.gov.uk/methodology/geography/geographicalproducts/ruralurbanclassifications)
+


This is great! :)

sofiapinto · 2023-10-25T11:03:17Z

asf_welsh_energy_consultation/getters/get_data.py

@@ -445,3 +445,19 @@ def load_wales_hp(wales_epc):
    wales_hp = wales_epc.loc[wales_epc.HP_INSTALLED].reset_index(drop=True)

    return wales_hp
+
+
+def pc_to_coords_df():


missing the dostrings :)
I've been finding it helpful to have the autodosctring extension in VSCode. It might also exist for your code editor.

sofiapinto · 2023-10-25T11:04:17Z

asf_welsh_energy_consultation/utils/utils.py

@@ -0,0 +1,9 @@
+def arial():


missing docstring for this function. Why not use Averta?

sofiapinto · 2023-10-25T11:04:33Z

requirements.txt

@@ -7,6 +7,7 @@ matplotlib
 odfpy
 selenium==4.2.0
 argparse==1.4.0
+keplergl


would it be better to set the version?

sofiapinto · 2023-10-25T11:09:52Z

asf_welsh_energy_consultation/pipeline/process_data.py

@@ -277,6 +278,36 @@ def get_mcs_retrofits():
    return mcs_retrofits


+def generate_hex_counts(wales_df, pc_df):


or something similar.. just changing names to make it more obvious what they represent. not required, just a suggestion :)

Suggested change

def generate_hex_counts(wales_df, pc_df):

def generate_hex_counts(wales_epc, location_info):

sofiapinto · 2023-10-25T11:10:19Z

asf_welsh_energy_consultation/pipeline/process_data.py

+    wales_df_coords = pd.merge(
+        wales_df, pc_df, on=["POSTCODE"]
+    )  # merge EPC with postcode df
+    wales_df_hex = add_hex_id(wales_df_coords, 6)  # add H3 hex id to each row


what does the 6 mean?

sofiapinto · 2023-10-25T11:49:39Z

asf_welsh_energy_consultation/pipeline/plotting.py

+
+def plot_kepler_graph(base_data, filename):
+    hex_map = KeplerGl(height=500)
+    hex_map.add_data(


To define the ranges appearing in the map i think you can do something like this:

# Define the custom color scale custom_color_scale = { "name": "Custom", # You can use any name you prefer "type": "sequential", "domain": [0, 0.1, 0.5, 1], # Change this to whatever you want :) "colors": ["#FF0000", "#FFFF00", "#00FF00", "#0000FF"] # Define colors for each range } # Add data with the custom color scale hex_map.add_data( data=base_data[["perc_true", "hex_id"]], name="Heat pump proportions", color_scale=custom_color_scale )

sofiapinto · 2023-10-25T14:23:55Z

asf_welsh_energy_consultation/pipeline/plotting.py

+def plot_kepler_graph(base_data, filename):
+    hex_map = KeplerGl(height=500)
+    hex_map.add_data(
+        data=base_data[["perc_true", "hex_id"]], name="Heat pump proportions"


one way to go around the name that appears on the plot is to rename "perc_true" before plotting.

sofiapinto · 2023-10-25T14:26:59Z

asf_welsh_energy_consultation/pipeline/plotting.py

+        file_name=os.path.join(fig_output_path["english"], f"{filename}.html")
+    )
+
+    print("Saved: " + filename + ".html")


this is saving to the english folder. is that what you want?

sofiapinto · 2023-10-25T14:43:49Z

Something else I remembered is that we might want to be strict and only show percentages when there's enough data for a specific hexagon, e.g. only show data for hexagon if denominator (of %) is above X (where X needs to be pre-set by us). I think we have enough data for each hexagon, but maybe worth checking. Hopefully it's not too much trouble - just checking the values of "total" for each hexagon.

crispy-wonton and others added 29 commits September 21, 2023 17:54

update README

cfe2149

improve and correct set up instructions and update output file extensions to html.

update local_data_dir in getters

aec3c3e

update local_data_dir to use relative file path from config base.yaml so config file can be updated by users rather than editing scripts.

update requirements.txt with versions

a7be30d

change outputs to .html files

60b59bc

change output graphs from produce_plots.py from .png to .html files to avoid altair CalledProcessError

fix introduced typo in requirements.txt

a56f765

update README

a37451a

update instructions in README to explain new way of running script and to record batches for historical analyses

update requirements.txt with argpase

43fd319

merge asf_senedd_response wales_analysis.py

cd6d1eb

add script to generate stats and charts from asf_senedd_response into produce_plots.py

add global parameters to config file

ba2dc7a

merge asf_senedd_response getters from loading.py

ecebb72

merge getters from loading.py in asf_senedd_response into get_data.py script

update function name

3c5d597

update EPC getter function name to match updates in get_data.py

add new files from asf_senedd_response repo

8768231

add files: - translation_config for welsh translations - plotting.py for plotting functions - augmenting.py for data processing and augmentation

resolve merge conflits - merge branch 'dev' into 03_merge_asf_senedd_…

465e997

…repo

update config variable name in __init__.py

31cf6d1

update to prevent module not subscriptable error

fix minor errors in produce_plots

bfe9ea3

use new config_file variable name

3a765f0

update domain min and max variables in time_series_comparison

fd5d6e6

improve selection of min and max dates

update README with new output files

7518c22

update plot max dates for newest EPC/MCS batches

1f29a92

add supp_data arg

97899cd

allows user to specify which directory with supplementary data to use, meaning new supplementary data can be added to analysis as it's updated

add function to get postcodes to coords

c93f5a1

add functions to produce kepler map

9779cff

update README with supp_data arg and historical analyses

a35e03d

delete formatting file and merge augmenting with process_data.py

b2d9361

add logging and improve how default domain min and max is determined

f79bd33

update import statements to remove import *

84694fa

fix merge conflicts - merge branch '03_merge_asf_senedd_repo' into 08…

a909fd3

…_add_map

fix import errors

f8177ed

crispy-wonton requested a review from sofiapinto October 17, 2023 09:05

sofiapinto reviewed Oct 25, 2023

View reviewed changes

Base automatically changed from 03_merge_asf_senedd_repo to dev October 30, 2023 16:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

08 add map #9

08 add map #9

crispy-wonton commented Oct 17, 2023 •

edited

Loading

sofiapinto left a comment •

edited

Loading

sofiapinto Oct 25, 2023

sofiapinto Oct 25, 2023

sofiapinto Oct 25, 2023

sofiapinto Oct 25, 2023

sofiapinto Oct 25, 2023

sofiapinto Oct 25, 2023 •

edited

Loading

sofiapinto Oct 25, 2023

sofiapinto Oct 25, 2023

sofiapinto Oct 25, 2023

sofiapinto Oct 25, 2023

sofiapinto commented Oct 25, 2023

		@@ -277,6 +278,36 @@ def get_mcs_retrofits():
		return mcs_retrofits


		def generate_hex_counts(wales_df, pc_df):

	def generate_hex_counts(wales_df, pc_df):
	def generate_hex_counts(wales_epc, location_info):

08 add map #9

Are you sure you want to change the base?

08 add map #9

Conversation

crispy-wonton commented Oct 17, 2023 • edited Loading

Instructions for Reviewer

Checklist:

sofiapinto left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sofiapinto Oct 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sofiapinto commented Oct 25, 2023

crispy-wonton commented Oct 17, 2023 •

edited

Loading

sofiapinto left a comment •

edited

Loading

sofiapinto Oct 25, 2023 •

edited

Loading