Jupyter docker: new full build with latest of everything

[tlvu]

Overview

New full build with latest of everything.

Changes

• New: save output of conda env export and environment size directly in the docker image for ease of tracking change between releases.
• Updated pull request template for new docker release.
• Jupyter env changes:
  • Unpin libnetcdf (Nb 3 raise wierd warning errors with newer Jupyter env PAVICS-landing#66 fixed).
  • Avoid dask != 2024.11.0 due to bugs with Raven nb.
  • Relevant changes (alphabetical order):

-  - bokeh=3.4.1=pyhd8ed1ab_0    
+  - bokeh=3.5.2=pyhd8ed1ab_0          

-  - cartopy=0.23.0=py311h320fe9a_0
+  - cartopy=0.24.0=py311h7db5c69_0

-  - cf_xarray=0.9.0=pyhd8ed1ab_0
+  - cf_xarray=0.10.0=pyhd8ed1ab_0

-  - cfgrib=0.9.11.0=pyhd8ed1ab_0      
+  - cfgrib=0.9.14.1=pyhd8ed1ab_0

-  - cftime=1.6.3=py311h1f0f07a_0
+  - cftime=1.6.4=py311h9f3472d_1   
   
-  - climpred=2.4.0=pyhd8ed1ab_0 
+  - climpred=2.5.0=pyhd8ed1ab_0   

-  - clisops=0.13.0=pyhca7485f_0 
+  - clisops=0.14.1=pyhd8ed1ab_0  

-  - dask=2024.5.0=pyhd8ed1ab_0   
+  - dask=2024.10.0=pyhd8ed1ab_0     

-  - esmf=8.4.0=nompi_hdb2cfa9_4      
+  - esmf=8.6.1=nompi_h4441c20_3   

-  - fiona=1.9.1=py311h3f14cef_0      
+  - fiona=1.9.5=py311hf8e0aa6_2    

-  - gdal=3.6.2=py311hadb6153_6          
+  - gdal=3.8.5=py311hf92cf48_11  

-  - geopandas=0.14.4=pyhd8ed1ab_0
+  - geopandas=1.0.1=pyhd8ed1ab_1      

-  - hvplot=0.9.2=pyhd8ed1ab_0        
+  - hvplot=0.11.1=pyhd8ed1ab_0    

-  - libnetcdf=4.8.1=nompi_h261ec11_106
+  - libnetcdf=4.9.2=nompi_h135f659_114

-  - numba=0.59.1=py311h96b013e_0     
+  - numba=0.60.0=py311h4bc866e_0        

-  - numpy=1.24.4=py311h64a7726_0 
+  - numpy=1.26.4=py311h64a7726_0  

-  - owslib=0.28.1=pyhd8ed1ab_0      
+  - owslib=0.32.0=pyhd8ed1ab_0       

-  - pandas=2.1.4=py311h320fe9a_0  
+  - pandas=2.2.3=py311h7db5c69_1     

-  - panel=1.4.2=pyhd8ed1ab_0       
+  - panel=1.5.3=pyhd8ed1ab_0         

-  - pydantic=2.7.1=pyhd8ed1ab_0
+      - pydantic==2.7.4                                                       

-  - pyogrio=0.5.1=py311h3f14cef_0                                                    
+  - pyogrio=0.7.2=py311hf8e0aa6_1 

-  - python=3.11.6=hab00c5b_0_cpython               
+  - python=3.11.10=hc5c86c4_3_cpython              

-  - rasterio=1.3.6=py311h567e639_0                 
+  - rasterio=1.3.10=py311h239598e_2                

-  - raven-hydro=0.2.4=py311h64a4d7b_0              
+  - raven-hydro=0.3.2=py311h81cb690_1              

-      - ravenpy==0.13.1     
+  - ravenpy=0.16.0=pyhd8ed1ab_0                                                                         

-  - rioxarray=0.15.5=pyhd8ed1ab_0                  
+  - rioxarray=0.17.0=pyhd8ed1ab_0                         

-  - roocs-utils=0.6.8=pyhd8ed1ab_0                        
+  - roocs-utils=0.6.9=pyhd8ed1ab_0                        

-  - scipy=1.13.0=py311h517d4fd_1                          
+  - scipy=1.14.1=py311he9a78e4_1                          

-  - shapely=2.0.1=py311h0f577a2_0                         
+  - shapely=2.0.4=py311h0bed3d6_1                         

-  - xarray=2023.8.0=pyhd8ed1ab_0                          
+  - xarray=2024.9.0=pyhd8ed1ab_1                          

-  - xclim=0.47.0=py311h38be061_0                          
+  - xclim=0.53.2=pyhd8ed1ab_0                                                 

-  - xesmf=0.8.5=pyhd8ed1ab_0                                                                                          
+  - xesmf=0.8.8=pyhd8ed1ab_0                                                  

-  - xscen=0.8.2=pyhd8ed1ab_0                                                  
+  - xscen=0.10.1=pyhd8ed1ab_0                                                 

Test

• Deployed as "beta" image in production for bokeh visualization performance regression testing.
• Manual test notebook https://github.com/Ouranosinc/PAVICS-landing/blob/master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb for bokeh visualization performance and it looks fine.
• Jenkins build:
  • Default notebooks, all passed: https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/f8c1c585a4f1b17c0a3dc67453695deb2c30fb11/docker/saved_buildout/jenkins-buildlogs-default.txt
  • Raven notebooks, only known HydroShare_integration.ipynb: https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/0b3419bba4aa42ee5a1fa9a8de0c8a6b91bf8547/docker/saved_buildout/jenkins-buildlogs-raven.txt

Related Issue / Discussion

• Matching notebook fixes:

  • Pavics-sdi: Adapt to cf-xarray changes, update pre-commit pavics-sdi#336
  • Finch: PR url
  • PAVICS-landing: Update ensemble notebook to use latest xclim PAVICS-landing#98
  • RavenPy: Update notebooks to work with changes to dask chunking, adapt to new Raven code conventions CSHS-CWRA/RavenPy#395
  • (...)

• Deployment to PAVICS: Jupyter: update to release py311-241111 bird-house/birdhouse-deploy#475

• Jenkins-config changes for new notebooks: PR url

• Other issues found while working on this one

  • Issue 1 URL
  • Issue 2 URL
  • (...)

• Previous release: Jupyter docker: new full build with latest of almost everything except xclim and ravenpy to smooth transition #121

Additional Information

Full diff conda env export:
release-py311-240506-update240508...f8c1c58

Full new conda env export:
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/0b3419bba4aa42ee5a1fa9a8de0c8a6b91bf8547/docker/saved_buildout/conda-env-export.yml

DockerHub build log
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/0b3419bba4aa42ee5a1fa9a8de0c8a6b91bf8547/docker/saved_buildout/docker-buildlogs.txt

[tlvu]

@tlogan2000 The new Jupyter env is live on PAVICS. You can notify users to restart their servers to get the new version.